Recipe Manager Rebuild

Filed in General Leave a comment

I’ve decided to rebuild and modernize the Recipe Manager application using NodeJS, MongoDB, and AngularJS for the UI. I’m also testing out Bootstrap 3 to see how well it handles stacking for mobile devices. So far I am quite impressed, at least on Android. I’ve also moved the project out of my SVN repos and into my GitHub: https://github.com/rhythmicdevil/recipe_manager

Simple RBAC using MongoDB and CakePHP

Filed in Development Leave a comment

I was tasked with implementing RBAC for a CakePHP application. After setting up CakePHP’s ACL I decided that MongoDB might be able to allow for an easier solution than MySQL. This is the first iteration and can probably stand for some improvement.

Requirements

Step 1
Create a MongoDB database
Create two collections, groups, users

Step 2
Create Models and Controllers for the collections.

Group Model

class Group extends AppModel
{
 
    public $displayField = 'group';
    public $hasMany = array('User');
    var $mongoSchema = array(
        'group' => array('type' => 'string'),
        'controllers' => array('type' => 'array'),
        'defaultController' => array('type' => 'string'),
        'defaultAction' => array('type' => 'string'),
        'created' => array('type' => 'datetime'),
        'modified' => array('type' => 'datetime')
    );
    public $validate = array(
        'group' => array(
            'isUnique' => array(
                'rule' => 'isUnique',
                'required' => true,
                'message' => 'Duplicate Group',
            ),
            'notEmpty' => array(
                'rule' => 'notEmpty',
                'required' => true,
                'message' => 'Group name is required',
            )
        )
    );
 
    /**
     * 
     */
    public function afterDelete()
    {
        parent::afterDelete();
        $this->deleteGroupFromUsers();
    }
 
    /**
     * This method searches all the User's group array for the current Group's ID and then 
     * deletes that record. This is typically done after deleting a group to prevent
     * orphan IDs in the User's account.
     */
    public function deleteGroupFromUsers()
    {
        $users = $this->User->find('all');
        foreach ($users as &$user)
        {
            foreach ($user['User']['groups'] as $i => &$group_id)
            {
                if ($group_id == $this->id)
                {
                    unset($user['User']['groups'][$i]);
                    $this->User->save($user);
                }
            }
        }
    }
 
}

User Model

class User extends AppModel
{
 
    public $belongsTo = array('Group');
    public $displayField = 'username';
    public $mongoSchema = array(
        'username' => array('type' => 'string'),
        'password' => array('type' => 'string'),
        'groups' => array('type' => 'array'),
        'created' => array('type' => 'datetime'),
        'modified' => array('type' => 'datetime')
    );
    /*
    public $validate = array(
        'username' => array(
            'isUnique' => array(
                'rule' => 'isUnique',
                'required' => true,
                'message' => 'Duplicate username'
            )
        ),
        'password' => array(
            'isUnique' => array(
                'rule' => 'isUnique',
                'required' => true,
                'message' => 'Password is required'
            )
        ),
        'groups' => array(
            'notEmpty' => array(
                'rule' => 'notEmpty',
                'required' => true,
                'message' => 'You must choose at least one group'
            )
        )
    );*/
 
}

Groups Controller

class GroupsController extends AppController
{
 
    /**
     *
     * @var type 
     */
    public $name = 'Groups';
 
    /**
     *
     * @var type 
     */
    public $paginate = array(
        'limit' => 15,
        'order' => array(
            'Group.group' => 'asc'
        )
    );
 
    public function __construct()
    {
        parent::__construct();
    }
 
    function beforeFilter()
    {
        parent::beforeFilter();
    }
 
    public function index()
    {
        $results = $this->paginate('Group');
        $this->set(compact('results'));
    }
 
    /**
     * add method
     *
     * @return void
     * @access public
     */
    public function add()
    {
        if (!empty($this->data))
        {
            $this->_cleanupPerms();
            $this->Group->create();
            if ($this->Group->validates())
            {
                if ($this->Group->save($this->data))
                {
                    $this->flash(__('Group Created.', true), array('action' => 'index'));
                }
            }
        }
 
        $this->set('controllers', $this->get_controllers_actions());
    }
 
    public function edit($id = null)
    {
        if (!$id && empty($this->data))
        {
            $this->flash(__('Invalid Group', true), array('action' => 'index'));
        }
 
        if (!empty($this->data))
        {
            $this->_cleanupPerms();
            // if ($this->Group->validates())
            //{
            if ($this->Group->save($this->data))
            {
                $this->flash(__('The Group has been saved.', true), array('action' => 'index'));
            }
            else
            {
                // @TODO need to handle when group is not saved
            }
            //}
        }
 
        if (empty($this->data))
        {
            $this->data = $this->Group->read(null, $id);
        }
 
        $this->set('controllers', $this->get_controllers_actions());
    }
 
    public function delete($id = null)
    {
        // @TODO Dont forget to come up with a cleaning function to purge deleted groups from the Users collection
        if (!$id)
        {
            $this->flash(__('Invalid Group', true), array('action' => 'index'));
        }
        if ($this->Group->delete($id))
        {
            $this->flash(__('Group deleted', true), array('action' => 'index'));
        }
        else
        {
            $this->flash(__('Group deleted Fail', true), array('action' => 'index'));
        }
    }
 
    /**
     * Uset any permissions that are set to 0. These come from checkboxes so 
     * possible values are 1 - selected or 0 - not selected
     * 
     * Unset any controllers that have no perms selected
     */
    protected function _cleanupPerms()
    {
        foreach ($this->data['Group']['controllers'] as $controller => &$perms)
        {
            foreach ($perms as $action => $perm)
            {
                if ($perm == 0)
                {
                    unset($perms[$action]);
                }
            }
            // If there are no perms in the controller then unset that bitch.
            if (empty($this->data['Group']['controllers'][$controller]))
            {
                unset($this->data['Group']['controllers'][$controller]);
            }
        }
    }
 
}

Users Controller

class UsersController extends AppController
{
 
    /**
     *
     * @var type 
     */
    public $name = 'Users';
 
    /**
     *
     * @var type 
     */
    public $paginate = array(
        'limit' => 15,
        'order' => array(
            'User.username' => 'asc'
        )
    );
 
    public function __construct()
    {
        parent::__construct();
    }
 
    function beforeFilter()
    {
        parent::beforeFilter();
    }
 
    public function index()
    {
        $results = $this->paginate('User');
        $this->set(compact('results'));
    }
 
    function login()
    {
        if ($this->Auth->login($this->data))
        {
            /*
             * You must do this to load the ACL data into session or the user
             * will not have permission to do anything.
             */
            $this->SimpleAcl->loadAcl();
            $this->redirect($this->Auth->loginRedirect, null, false);
        }
    }
 
    function logout()
    {
        $this->Session->destroy();
        $this->redirect($this->Auth->logout());
    }
 
    /**
     * add method
     *
     * @return void
     * @access public
     */
    public function add()
    {
        if (!empty($this->data))
        {
            if ($this->data['User']['password'] == $this->Auth->password($this->data['User']['password_confirm']))
            {
                $this->User->create();
 
                //if ($this->User->validates())
                //{
                if ($this->User->save($this->data))
                {
                    $this->flash(__('Account Created.', true), array('action' => 'index'));
                }
                // }
            }
        }
        $this->set('groups', $this->User->Group->find('list'));
    }
 
    public function edit($id = null)
    {
        if (!$id && empty($this->data))
        {
            $this->flash(__('Invalid User', true), array('action' => 'index'));
        }
 
        if (!empty($this->data))
        {
            //if ($this->User->validates())
            //{
            if ($this->User->save($this->data))
            {
                $this->flash(__('The User has been saved.', true), array('action' => 'index'));
            }
            else
            {
                // @TODO need to handle when group is not saved
            }
            // }
        }
 
        if (empty($this->data))
        {
            $this->data = $this->User->read(null, $id);
        }
 
        $this->set('groups', $this->User->Group->find('list'));
    }
 
    public function reset_password()
    {
 
    }
}

And finally the SimpleRBAC Component

/**
 * MongoDB RBAC/ACL
 * 
 * Expects that Cake's Auth component will handle logging the user in and creating
 * the User session.
 * 
 * To use this component you need two models Users, Groups that are tied to a 
 * MongoDB.
 * 
  class Group extends AppModel
  {
 
  public $displayField = 'group';
  public $hasMany = array('User');
  var $mongoSchema = array(
  'group' => array('type' => 'string'),
  'controllers' => array('type' => 'array'),
  'defaultController' => array('type' => 'string'),
  'defaultAction' => array('type' => 'string'),
  'created' => array('type' => 'datetime'),
  'modified' => array('type' => 'datetime')
  );
 
  }
 * 
  class User extends AppModel
  {
 
  public $belongsTo = array('Group');
  public $displayField = 'username';
  public $mongoSchema = array(
  'username' => array('type' => 'string'),
  'password' => array('type' => 'string'),
  'groups' => array('type' => 'array'),
  'created' => array('type' => 'datetime'),
  'modified' => array('type' => 'datetime')
  );
  }
 * 
 * You must have a UsersController and its login method should contain at least
 * this
 * 
  function login()
  {
  if ($this->Auth->login($this->data))
  {
 
  //You must do this to load the ACL data into session or the user
  //will not have permission to do anything.
 
  $this->SimpleAcl->loadAcl();
  $this->redirect($this->Auth->loginRedirect, null, false);
  }
  }
 * 
 *
 * You must have a GroupsController and it should contain the following method. This
 * method must be called before writing group information to the DB
 * 
 
  protected function cleanupPerms()
  {
  foreach ($this->data['Group']['controllers'] as $controller => &$perms)
  {
  foreach ($perms as $action => $perm)
  {
  if ($perm == 0)
  {
  unset($perms[$action]);
  }
  }
  // If there are no perms in the controller then unset that bitch.
  if(empty($this->data['Group']['controllers'][$controller]))
  {
  unset($this->data['Group']['controllers'][$controller]);
  }
 
  }
  }
 * 
 * 
 * Your AppController::beforeFilter should contain at least this
 * 
  if(!$this->SimpleAcl->check())
  {
  if(!$this->Session->check('Auth.User.defaultPath'))
  {
  $this->redirect($this->Auth->loginRedirect);
  }
  else
  {
  $this->redirect($this->Session->read('Auth.User.defaultPath'));
  }
  }
 * 
 */
class SimpleAclComponent extends Object
{
 
    public $components = array('Session', 'Auth');
 
    /**
     *
     * @var array
     */
    public $allowedActions = array('login', 'logout');
 
    /**
     *
     * @var string 
     */
    public $userModel = 'User';
 
    /**
     *
     * @var string
     */
    public $groupModel = 'Group';
 
    /**
     *
     * @var controller instance
     */
    public $controller;
 
    /**
     * 
     * 
     *
     * @param object $controller A reference to the instantiating controller object
     * @return void
     * @access public
     */
    function initialize(&$controller, $settings = array())
    {
        $this->controller = &$controller;
 
        if (is_array($settings))
        {
            foreach ($settings as $setting => $values)
            {
                // Only add properties that exist for the class
                // This prevents polution of the class
                if (property_exists(get_class($this), $setting))
                {
                    $this->$setting = $values;
                }
            }
        }
    }
 
    /**
     * Main execution method.  Handles redirecting of invalid users, and processing
     * of login form data.
     *
     * @param object $controller A reference to the instantiating controller object
     * @return boolean
     * @access public
     */
    function startup(&$controller)
    {
 
    }
 
    /**
     * Find the requested controller/action in Auth.User.AclPerms. If its not
     * found then redirect the user back to the referring URL
     * 
     * AclPerms is a list of Group
     * 
     * @return boolean
     */
    public function check()
    {
        //var_dump($this->Session->read('Auth.User.AclPerms'));
 
        /*
         * Check to see if we need to do any checking based upon the value
         * of Auth::AllowActions vs. the current request
         * 
         * If the current controller and action are in the allowed actions array
         * then return true
         * 
         */
        if (in_array(strtolower($this->controller->name), array_keys($this->allowedActions)))
        {
            if (in_array($this->controller->action, $this->allowedActions[strtolower($this->controller->name)]))
            {
                return true;
            }
        }
        /*
         * Find the requested controller/action in Auth.User.AclPerms
         */
        $currentController = $this->controller->name;
        $currentAction = $this->controller->action;
        if ($this->Session->check('Auth.User.AclPerms'))
        {
            foreach ($this->Session->read('Auth.User.AclPerms') as $group)
            {
                foreach ($group['Group']['controllers'] as $controller => $perms)
                {
                    if ($currentController == $controller)
                    {
                        foreach ($perms as $k => $v)
                        {
                            if ($k == $currentAction && $v == '1')
                            {
                                return true;
                            }
                        }
                    }
                }
            }
        }
        /*
         * Set a semaphore in session so that we can trigger a dialog box
         * after the redirect happens
         */
        $this->Session->write('rbacDenied', true);
        /**
         * Keep the user on the same page by redirecting them back to the referer.
         */
        $this->controller->redirect($this->controller->referer());
    }
 
    /**
     * Loads the Group permissions into the User object in session
     */
    public function loadAcl()
    {
        $aclPerms = array();
        /*
         * Look for the groups array in the User session.
         * If it does not exists thrown an error
         */
        if (!$this->Session->check('Auth.User.groups'))
        {
            $this->cakeError('simpleAclError', array('msg' => 'No groups found in user record'));
        }
 
        /*
         * Look for the simpleAcl element in the User object in session.
         * If its not there then get it and insert it into the record
         * If you cant get it throw an error.
         */
        if (!$this->Session->check('Auth.User.AclPerms'))
        {
            foreach ($this->Session->read('Auth.User.groups') as $group_id)
            {
                $params = array(
                    'conditions' => array(
                        '_id' => $group_id
                    ),
                    'fields' => array(
                        'controllers' => 1,
                        'group' => 1,
                        'defaultController' => 1,
                        'defaultAction' => 1
                    )
                );
 
                $aclPerms[] = $this->controller->User->Group->find('first', $params);
            }
 
            if (empty($aclPerms))
            {
                $this->cakeError('simpleAclError', array('msg' => 'No groups found'));
            }
            /*
             * If the User is in more than one group the default and controller
             * is the root of the application. Otherwise redirect the user to
             * the group's default controller/action
             * 
             */
            if (count($aclPerms) == 1)
            {
                $c = $aclPerms[0]['Group']['defaultController'];
                $a = $aclPerms[0]['Group']['defaultAction'];
                $this->Session->write('Auth.User.defaultPath', array('controller' => $c, 'action' => $a));
                /*
                 * A one time redirect for after login
                 */
                $this->Auth->loginRedirect = array('controller' => $c, 'action' => $a);
            }
        }
 
        $this->Session->write('Auth.User.AclPerms', $aclPerms);
    }
 
    /**
     * Component shutdown.  If user is logged in, wipe out redirect.
     *
     * @param object $controller Instantiating controller
     * @access public
     */
    function shutdown(&$controller)
    {
 
    }
 
}

Add the following to your AppController

    public $components = array(
        'SimpleAcl' => array(
            'allowedActions' => array(
                'users' => array(
                    'login', 'logout'
                )
            )
        ),
        'RequestHandler',
        'Session',
        'Auth' => array(
            'loginRedirect' => array('controller' => 'main', 'action' => 'index'),
            'autoRedirect' => false
        )
    );

Add the following to the AppController::beforeFilter()

public function beforeFilter()
    {
        parent::beforeFilter();
        Security::setHash('md5');
        //$this->Auth->allow('*');
 
        /*
         * Check the user's request against the group's loaded permissions. If
         * this fails the user will be redirected to the referring URL
         */
        $this->SimpleAcl->check();
        /*
         * Check to see if the semaphore has been set and if its true. If so
         * show the user a dialog box.
         */
        if ($this->Session->check('rbacDenied') && $this->Session->read('rbacDenied') == true)
        {
            $this->Session->delete('rbacDenied');
            $this->dialogMessage(
                    array(
                        'title' => 'No Access',
                        'status' => 'warning',
                        'msg' => 'You dont have access to that.'
                    )
            );
        }
 
    }

SAMBA Setup on CentOS 6.x

Filed in Dev Tools Leave a comment

Just some notes I took when installing SAMBA on CentOS 6.x. for a development environment. DO NOT use this setup for any sort of production system, this appropriate for development only. I run CentOS VMs on my Windows 7 box for development work and I like to map my home directory for each to Windows for ease of access and editing of files.

Turn off SELinux

vi /etc/selinux/config

Set the following: SELINUX=disabled

Turn off IPTables

chkconfig iptables off

Restart the Machine

shutdown -r now

Install SAMBA

yum install samba samba-client

Update Configuration File

vi /etc/samba/smb.conf

Edit the network section

[global]
 # ----------------------- Network Related Options -------------------------
 #
 # workgroup = NT-Domain-Name or Workgroup-Name, eg: MIDEARTH
 #
 # server string is the equivalent of the NT Description field
 #
 # netbios name can be used to specify a server name not tied to the hostname
 #
 # Interfaces lets you configure Samba to use multiple interfaces
 # If you have multiple network interfaces then you can list the ones
 # you want to listen on (never omit localhost)
 #
 # Hosts Allow/Hosts Deny lets you restrict who can connect, and you can
 # specifiy it as a per share option as well
 #
 workgroup = 
 server string = Samba Server Version %v
 #netbios name = SERVER
 ; interfaces = lo eth0 192.168.12.2/24 192.168.13.2/24
 interfaces = lo eth0
 ; hosts allow = 127. 192.168.12. 192.168.13.
 ; hosts allow = 127. 192. 10.

Create a section for your share

#============================ Share Definitions ==============================

[homes]
        comment = Home Directories
        browseable = yes
        writable = yes
;       valid users = %S
;       valid users = MYDOMAIN\%S

[swright]
        path=/home/swright
        writeable=yes
        browseable=yes
        valid users=swright

Set your SAMBA password

I usually just make this the same password as my Linux user account

smbpasswd -a swright

Restart the SAMBA Services

service smb restart

service nmb restart

Test the Config File

testparm

You should see something like:

[root@localhost ~]# testparm
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
Processing section "[homes]"
Processing section "[printers]"
Processing section "[swright]"
Loaded services file OK.
Server role: ROLE_STANDALONE
Press enter to see a dump of your service definitions

[global]
        workgroup = WORKGROUP
        server string = Samba Server Version %v
        interfaces = lo, eth0
        log file = /var/log/samba/log.%m
        max log size = 50
        idmap config * : backend = tdb
        cups options = raw

[homes]
        comment = Home Directories
        read only = No

[printers]
        comment = All Printers
        path = /var/spool/samba
        printable = Yes
        print ok = Yes
        browseable = No

[swright]
        path = /home/swright
        valid users = swright
        read only = No

Issues

One big issue that I run into is due to some conflicts with Network Manager in CentOS 6. NMB will try to start before the network is ready and so fails. Here is a thread about it: http://forums.fedoraforum.org/archive/index.php/t-290347.html

I just shut off the Network Manager using the GUI. I am sure there is a better way to do it though. Will post later.

Mackie MR5 Main Driver Replacement

Filed in Studio Leave a comment

Here is a step by step guide on how to replace the main driver in a Mackie MR5 speaker enclosure.

 

01. Take out the 6 perimeter screws on the back

Step 1

 

02. Remove the back plate

Step 2

 

03. Unplug connector at CN2 that has 1 red and 1 black wire
04. Unplug connector at CN5 that has 1 blue and 1 white wire
05. Unplug connector at CN4 that has 2 white wires (I used needle nosed pliars)

Steps 3, 4, 5

 

06. Remove bass port
07. Take out the 8 perimeter screws on the inner baffle
08 Slide out the inner baffle catty cornered horizontally and then lay it down partially inside
because the wires are too short and glued to the baffle to remove it all the way.

Step 8

 

09. Looking through the back at the back side of the front face locate the 6 screws remove them with a very long phillips head

Step 9

 

10. There are two more screws on the inner surface of the top of the box. Remove them with a very short phillips head

11. Slide the inner baffle back in so the wires dont stretch when you take off the front face

12. Carefully slide the front face off. The tweeter wires and “on” light are connected through the face so will only get about an inch off the box.

13. Rotate the front face carefully so you can lay it on the top of the box

Step 13

 

14. Remove the four screws holding the driver.

15. Carefully pull out the driver

16. Unplug the red wire from the positive terminal.

17. Unplug the black from the negative terminal and remove the driver

Step 15

 

Reverse order to put it back together. Make sure to snug the screws without over tightening.

 

,

RSA Token Generator

Filed in Dev Tools | Development Leave a comment

I recently switched my desktop OS to CentOS 6 from Windows 7. One of the issues I ran into was generating an RSA token. I followed these instructions:
http://www.flowerchild.org.uk/archive/2011/01/13/getting-the-rsa-software-key-working-on-android.aspx

They worked great but I could only get the token converter to work on CentOS 5. It had something to do with the linker but I did not investigate to much further once I got it working. You only have to do it once so I was not that concerned.

jQuery Datatables Plug-in

Filed in jQuery Leave a comment

I had to create some editable datatables and decided to give Datatables.net a try. It has tons of functionality and some great features. I read the documentation and tried a couple of test implementations and it seemed to work quite well. However, and this may be because of the way I think, it seemed to be missing a crucial concept when it comes to have many datatables. I DONT want to have to write an initialize block for every datatable that may happen to appear in my application. Thats a totally crap way of coding in my opinion.

You need to install the Datatables and Jeditable plugins for jQuery.

Here is how I defined the solution:

A table should become editable simply by adding some CSS classes and a couple of special attributes. The reason for all this is that I want to be able to define all the stuff for the table in one place.

  1. The table must have a unique ID
  2. The table must have a custom attribute off  ’editURL’ with a value of the URL where the edit data will be sent
  3. Any column to be edited must have a class of ‘editable’
  4. The row containing the editable column must have an ID that the columns content can be tied to.
  5. The table must have a footer that contains a hidden row which is the template for new rows.

I know that custom HTML attributes can be a problem but I will use them in this case for two reasons. One is that the Datatables plugin adds it’s own and two this is a closed audience application so I have complete control.

Here is an example of the table HTML:

<table id="botnetDomainList" class="editable" editurl="/botnets/edit_domain">
    <thead>
        <tr>
	    <th>Domain</th>
	    <th class="control>Delete</th>
	</tr>
    </thead>
    <tbody>
        <tr id="cheeseburger.com">
	    <td class="editable ">cheeseburger.com</td>
	    <td><div class="deleteRow"></div></td>
	</tr>
    </tbody>
    <tfoot>
        <tr style="display:none;" class="template">
	    <td class="editable" style="">edit</td>
	    <td>
		<div class="deleteRow"></div>
	    </td>
	</tr>
	<tr>
	    <td></td>
	    <td>
	        <div class="addRow"></div>
	    </td>
	</tr>
    </tfoot>
</table>

Here is the Javascript that supports the tables. Once again my goal was to have to only write one Datatables Init function for all datatables. It may not cover some edge cases but it should work for majority. Read the comments to get details of what I did and why.

    $.add_field_edit_handlers = function(oTable){
        /*
             * Setup up the editable columns.
             * --The table must have an ID.
             * --The table must have a custom attribute of 'editURL' containing the url of
             *   where to send the edit data.
             * --The columns to be edited must have the class 'editable'
             * --The row containing the editable columns must have some sort of
             *   identifying information.
             */
        $('#' + oTable.attr('id') + ' td.editable').editable($(oTable).attr('editURL'), {
            "callback": function( sValue, y ) {
                /*
                 * Handle return from the server
                 */
                var aPos = $(oTable).dataTable().fnGetPosition( this );
                $(oTable).dataTable().fnUpdate( sValue, aPos[0], aPos[1] );
            },
            "submitdata": function ( value, settings ) {
                /*
                     * Sends to the server
                     */
                return {
                    "row_id": this.parentNode.getAttribute('id'),
                    "column": $(oTable).dataTable().fnGetPosition( this )[2]
                };
            }
        });
    }
    /**
     * Setup a generic editable datatable
     */
    $('table.editable').dataTable({
        "bPaginate": false,
        "bLengthChange": false,
        "bFilter": false,
        "bSort": false,
        "bInfo": false,
        "bAutoWidth": false,
        "fnInitComplete" : function(oSettings, json){
            var oTable = this;
            $.add_field_edit_handlers(oTable);
        }
    });
    /**
     * Add a click event handler to insert the hidden row template in the footer
     * into the tables body. This is sort of a pain in the ass due to the way that
     * the Datatables plugin works. You cant simply add your row. Datatables adds
     * rows by taking an array of cell content, not the cells themselves. This
     * means that our template row's cells lose their classes. So we have to extract
     * the classes and then add them back once Datatables.fnAddData() has added
     * the new row. swright 02-21-2012
     */
    $('.addRow').click(function(){
        // Get the ID of the clicked table
        var oTable = $(this).closest('table');
        // Something to hold the template row
        var template = new Array();
        // Something to hold the styles from each cell in the template row
        var td_class = new Array();
        /*
         * Iterate the TDs in the template row. Add the content for each cell to
         * the template array. Add the CSS class for each cell to the td_class array
         */
        $(oTable).find('tfoot tr.template td').each(function(){
            template.push($(this).html());
            if(typeof($(this).attr('class')) != "undefined"){
                td_class.push($(this).attr('class'));
            }
        });
        // Add the template array
        $(oTable).dataTable().fnAddData(template);
        /*
         * Iterate the new row's TDs and add the classes that we saved in the
         * td_class array.
         */
        $(oTable).find('tbody tr:last td').each(function(index, val){
            if(typeof(td_class[index]) != "undefined"){
                $(val).attr('class', td_class[index]);
            }
        });
        // Added the field edit handlers
        $.add_field_edit_handlers(oTable);
    });

Favorite Linux Tools and Other Stuff

Filed in Dev Tools | Linux Leave a comment

This is just a list of some useful Linux stuff I need to reference from time to time.

Linux Screen is an excellent tool for grouping multiple consoles into one SSH window.

 

Ever have Putty lockup on you? You probably hit CTRL-S while using Screen. Here is the solution.

MongoDB Training

Filed in Development | MongoDB Leave a comment

These are some notes from my recent MongoDB training session with 10gen’s Richard Kreuter.

Lack of Schema

MongoDB does not impose or require any schema for the documents you store in a collection. The application must enforce any schema requirements, for instance when inserting a user record you may require the presence of at least a username field. The application must validate this before writing the record. Its possible to add and subtract elements from the document as the application’s needs change over time. This is both good and bad. Compared to an RDMS adding columns is a snap, but your application must be able to deal with the fact that it may get documents in different formats from the same collection. An excellent way to deal with this is to use a version number field in your documents. The application will know what document version number its dealing with and you can add routines to upgrade any documents you find while doing other work such as reads, and or updates. Or you could just have a little worker program that upgrades documents to the newest specs in the background. The point is you have lots of flexibility but that comes with some management overhead.

Document Padding

Document padding is the idea of adding extra blank data to your documents. For instance if you have User document that contains an array of Roles that the User is assigned to. When store the document initially the User may only have one Role and so you store an array of one item. As the User gets more Roles the document’s array will grow. The problem is that the documents are stored on disk. If the document grows beyond the size allocated on the disk the document must be moved to a larger space. This is an expensive operation. A good strategy is to pad the Roles array to a reasonable size with empty values to reduce the possibility of disk relocation of the record.

To Join or not to Join

In an RDMS that has been normalized to contain no duplicate data you have to join your tables together in order to get a document. This means you dont have to worry about maintaining duplicate values. In MongoDB you generally want to store to documents as you would want to see them which means you may (probably will) have duplicate values. Take for instance the idea of a collection of IP addresses that contain domain information and a collection of domains that contain IP information. You would have both collections because you would want to index on different fields and quickly fetch data from both points of view but you would also have duplicate data. You also have the possibility that in between writing your domain info and then writing your IP info the DB dies in some ugly way. This would leave you in the bad position of having out of sync data. A good strategy to handle this is to create a third collection that holds a sort of “transaction”. Its not a transaction in the sense of an RDMS transaction, but I dont have a better term at the moment because my coffee has not kicked in yet. So what you do is create this third collection and store a document that describes what action you want to perform, add, update, or delete, the data you want to do it with, and a timestamp. Then your application can attempt to “play” the action as many times as it has to, hopefully once, to fulfill the original request. Once it completes successfully you remove the record from the “transaction” collection.

Replication and Sharding

To be clear, a replica set is a group of mongod processes that all contain the same document data whose purpose is to prevent data loss. At least one of the processes acts as a writeable server at any one time and is known as the primary node. In general you would run each replica on a separate piece of hardware. There’s not much point to having them reside on the same box excepting a development deployment and even then, VMs would be a better option. But failing that, each replica must have it’s own data directory, log file and unique tcp port to communicate on. Any modifications made to the replica set config file must be made on the primary node.

The OpLog

Every data manipulation query sent to a mongod process is collected in the op log. Once you have setup your replica sets you will see a collection in the local database called oplog. By default the oplog is 5% of the available disk space. However you can change this when starting mongod like so:

mongod --oplogSize 200 // in MB

So why would you want to do this? The OpLog is a capped collection which makes it work like queue. Once the collection gets to the cap size the oldest records are removed one by one as new records come in. Depending on the amount of write operations your database is handling and the speed at which the replicas can pick up those changes its possible, although unlikely, that your oplog would roll off stale data before the replicas had a chance to write the data. The replicas would no longer be able to sync after that point. A more likely scenario is that for some reason your replicas lose connectivity for some period of time that is greater than the period of time recorded in the oplog. When the replica comes back online it will not be able to sync. So its better to err on the side of a bigger oplog within reason.

Sharding

 (that joke is not as funny as you think)

Sharding is used to spread data and writing operations around. Its that simple. Well ok not really. Sharded clusters consists of several, at least 1, shards each of which is a replica set. I’ll say it again, a single shard is a replica set. There are some rules about shards that may or may not be intuitive given what you know about Mongo.

  1. In a sharded system every document in a collection must contain the same fields.
  2. You are required to have exactly 3 config servers that store the metadata and routing information for the cluster.
  3. You must have at least one but usually more mongos processes running. MongoS is the name of the binary not multiple mongo.
  4. This is what your application will connect to and it also does not have to reside on the database server. In fact I think its better if its resides on the application server.
  5. A shard key’s value is immutable! The only way to change the value is pull the document, make the change and then insert it as a new document a and delete the original.
  6. Shard key values dont have to be unique in a collection. This means that if you want to update a specific document you must include some other identifying field in addition to the shard key.
Mongos (n)
Shard 1 Shard 2 Shard 3
RS1
Primary
Secondary
Secondary
RS2
Primary
Secondary
Secondary
RS3
Primary
Secondary
Secondary

Following the replica set rules, each member of the replica sets should ideally reside on separate hardware. So in the example above you would have 9 physical pieces of hardware.

Picking a shard key is something you must consider carefully. While its possible to change a shard key in a running database its an incredibly painful process that will effectively require shutting down your database which is not easy in a production situation.

One reasonable strategy is to generate a hash and store that in the document as your shard key. Something else to watch out for are keys that are monotonic in nature. Because shards are split up in ranges based upon the values of the shard keys if you have a monotonic shard key you will end up always writing to one shard which means you lose write balancing in your cluster.

Assume your shard key is an integer and you have 3 shards. The shards will balance the number of records by splitting the data over the shards
based upon the values of the key.

Shard 1 Shard 2 Shard 3
key range: 1-10 key range: 11-20 key range: 21->infinity

So one of the shards is always going to contain the higher range of keys and because your key values are always increasing mongos is always going to write to that shard. At some point the data will split and balance but you will always be writing to one shard. So beware of monotonic keys.

This is an excellent reference source.

RabbitMQ, PHP, Centos5

Filed in Dev Tools | Development Leave a comment

Trying out RabbitMQ as a messaging system for modules written in different languages. The installation is a little bit tricky.  The target  is running in Virtual Box. I started with:

  • Centos 5.6 (64bit)
  • PHP 5.2.10
  • Rabbit MQ Server 2.6.1

The PHP module ampq is picky about which version of PHP it runs against. Centos 5 comes with PHP5.2 which ampq wont run compile with, So my first step was to upgrade PHP to the right version. Since I like to try and use yum for package management whenever possible I found a repo with the versions I was looking for. I added the repo by creating the following file:

/etc/yum.repo.d/webtatic.repo

And then adding the following content to it:

[webtatic]
name=Webtatic Repository
baseurl=http://repo.webtatic.com/yum/centos/5/x86_64/
gpgcheck=0

Once that is done you can then update the PHP version. I updated to the latest version that Webtatic had available which at the time was 5.3.8

yum update php*

The PHP update pooched my php.ini file and was incompatible with my Mongo module. The php.ini file no longer allows “#” as a comment marker anymore. To upgrade the Mongo module simply use PECL:

pecl upgrade mongo

Now you can finally install the ampq module:

pecl install amqp

Make sure to add a link to the amqp.so file

extension=amqp.so

MongoDB Sharding

Filed in Development | General Leave a comment

Today I had to create a MongoDB sharding setup on 4 VM servers.  One of the servers would be the primary shard and also contain the MongoS process and the MongoD configdb process. You can find out more about Mongo sharding here. There were some things that are not spelled out in the documentation and were not obvious to me so I figured I would list them here for reference.

Continue Reading

TOP