Chaining Sidekiq Workers

I’m working on an Rails app which syncs data with a 3rd party.  It turned out though that when one change was made in my application, it unfortunately needed to make 3 different sync requests to the 3rd party application.  And, the 3rd party app is finicky.  The request need to be in order, one at a time.

I needed to chain these requests together, but really did not want to introduce any of the gems I could find that would help me.  The gems were just “too much” for what I needed.  I ended up writing a “Command” class to call the various workers.  The command object would include itself as a param to the Sidekiq worker and the worker would then call the command class as part of it’s last step.  It ended up looking like this:

	require 'json'
	class AddIndividiualCmd
	attr_reader :roster_id, :current_step
	STEPS = %w(
	PhoneInsertWorker
	RosterLinkToSchoolWorker
	RosterLinkToSchoolAddressWorker
	)
	def initialize(roster_id, current_step = nil)
	@roster_id = roster_id
	@current_step = current_step || STEPS[0]
	end
	def run
	case @current_step
	when "PhoneInsertWorker"
	worker = Object.const_get(@current_step)
	roster = Roster.find(@roster_id)
	worker.perform_async(roster.phone.id, self.to_json)
	when "RosterLinkToSchoolWorker"
	worker = Object.const_get(@current_step)
	worker.perform_async(@roster_id, self.to_json)
	when "RosterLinkToSchoolAddressWorker"
	worker = Object.const_get(@current_step)
	worker.perform_async(@roster_id, self.to_json)
	else
	Rails.logger.info "Finished AddIndividiualCmd for roster [#{@roster_id}]"
	end
	end
	def next
	current_step_index = STEPS.index(@current_step)
	@current_step = current_step_index.nil? ? nil : STEPS[(current_step_index + 1)]
	run
	end
	def to_json(*a)
	{
	"json_class" => self.class.name,
	"data" => {'roster_id' => @roster_id, 'current_step' => @current_step}
	}.to_json(*a)
	end
	def self.json_create(o)
	new(o['data']['roster_id'], o['data']['current_step'])
	end
	end

view raw

AddIndividualCmd.rb

hosted with ❤ by GitHub

	class RosterLinkToSchoolWorker
	include Sidekiq::Worker
	include MyHelper
	def perform(roster_id, cmd_json = nil)
	roster = Roster.find(roster_id)
	link_roster_to_school(roster, cmd_json)
	end
	def link_roster_to_school(roster, cmd_json)
	my_helper_method(roster)
	if cmd_json.present?
	cmd = JSON.load(cmd_json)
	cmd.next()
	end
	end
	end

view raw

RosterLinkToSchoolWorker.rb

hosted with ❤ by GitHub

The command class needs to be able to serialize and deserialize from JSON since that is how Sidekiq persists parameters.  I wasn’t sure why I needed to do the deserialization in the worker myself, but it “worked”.

One thing I like about this strategy better than having a worker directly call the next worker is that from this command class, I can see exactly what workers are going to be called and in what order.  I can also use the workers independently, without being chained.  And, if I have another command  that needs to call a worker but getting the worker param is different, that logic is encapsulated in the command class.  Not spread around who knows where.

I still consider myself relatively new to Ruby and I’m not sure of the merits of Object.const_get vs. some other method, so feedback is appreciated.

Version Info:

• Rails 4.2.7
• Ruby 2.2.4
• Sidekiq 4.1.4

https://gist.github.com/aghuddleston/c3051a563117996a1e659fcef48812d6#file-addindividualcmd-rb

Rails, Ransack, Simple Form, Checkboxes and More

I’ve been working in Rails recently, with a search from using Ransack that has a bunch of checkboxes.  This is a quick summary of some “features” I figured out along the way, for future reference.

Sample Form

In HAML:

= search_form_for @q, :url => search_path, :html => { :method => :get } do |f|
  = f.input :custom_scope, label: false, as: :boolean, inline_label: "#{ t 'label.label_here'}",
      checked_value: true,
      input_html: {checked: @q.custom_scope == "true"}
  = f.input :contact_info_opt_in_true, as: :boolean, label: false, 
      inline_label: "#{t 'label.opt_in'}"
  = f.input :contact_info_birthday_notification_true, as: :boolean, 
      label: false, inline_label: "#{t 'label.birthday_notification'}"
  = f.input :contact_info_interests_in, label: "#{t 'label.interests'}",
      collection: Interest.all
  = f.submit t('search_btn'), :class => 'btn btn-success'
  = link_to t('label.search_clear'), request.path, class:"cancel-button", role: :button

Sample Controller

def index
 if params[:q].present?
   clear_boolean(params[:q], :contact_info_opt_in_true)
   clear_boolean(params[:q], :contact_info_birthday_notification_true)
   clear_boolean(params[:q], :custom_scope)
 end

 if (params[:q].present? && params[:q].reject { |k, v| v.is_a?(Array) ? ( v.length == 1 ): v.blank? }.present?)
   # At least one search criteria is required
   @q = ContactInfo.ransack(params[:q])

   # Find the records for the current page.  Will use the max pages if current page is beyond the limit.
   @contact_infos = @q.result(distinct: true)
                      .page(current_page).per(25)
 else
   # There are no search params
   @q = ContactInfo.ransack(nil)
   @contact_infos = Kaminari.paginate_array([]).page(current_page).per(25)
 end

end

def max_pages
 10
end

private

# Current page or the max page limit
def current_page
 [(params[:page] || 1).to_i, max_pages].min
end

def clear_boolean(q, condition)
 q.delete(condition) if q[condition] == "0"
end

Ignore Unchecked Checkbox

In my form, if the checkbox is checked I want to include that field in the search criteria, but if it is unselected I want to ignore it completely, not just find the “false” ones. In the controller, before processing the search, the checkbox params are cleared. You can see this in the controller with the call to “clear_boolean” before ransack is called.

Use a Checkbox to Search by a Scope

The checkbox with the name “custom_scope” will search by the custom scope if the boolean is checked. In order to get this to work I needed to:

1. Clear the checkbox value if it is unchecked (see above)
2. Setup the scope for ransack. In the model define the scope and then add the scope to the ransackable_scopes

     def self.ransackable_scopes(auth_object = nil)
       %i(custom_scope)
     end
   

3. Make the checkbox checked value equal to true. The value true, not the string “true”. You can see the setting in the checked_value of the custom_scope input.
4. Finally, to make the checkbox be selected when the page reloads after the search, I set the checked value equal to the search param value. You can see this in the input_html of the custom_scope input

Search Only When Some Criteria is Selected

One of the requirements was for the search page to not run the search by default, but required some search criteria. To enforce this, I checked if there were any search params before running the search. Since my search criteria included a multi-select box, it also had to inspect some array lengths. Example is in the controller code, when checking if params are present.

Max Pages

Finally, I had a requirement to only show a max number of pages, regardless of if the results exceeded that page limit. My app was using will_paginate, but I found that Kaminari supported this requirement better. The call to “current_page” will ensure the results do not go past that max. In addition, I had to “fake” the empty search results for the view to properly display. You can see code above with the call to Kaminari.paginate.array.

Lots of searching to pull this all together and unfortunately I no longer have the links to the various sources. Stack Overflow featured heavily.

Version Info:
Rails 4.2.6, Ruby 2.2.4
Simple Form 3.2.1
Ransack 1.7.0
Kaminari 0.17.0

Mocking Fog with CarrierWave and Minitest

I kept getting the  error

Expected(200) <=> Actual(404 Not Found)

while trying to use Fog.mock! in unit tests.  I followed the guidelines in Fog and Carrierwave, but no luck.  Here’s the trick:  Fog uses credentials to segregate mock data, so be sure the controllers and tests are using the same credentials.

Final setup:

carrierwave.rb

CarrierWave.configure do |config|
  s3_config = YAML::load(File.read(Rails.root.join('config','s3.yml')))[Rails.env]

  config.fog_credentials = {
    :provider => 'AWS',
    :aws_access_key_id => s3_config['access_key_id'],
    :aws_secret_access_key => s3_config['secret_access_key'],
  }
  config.fog_directory = s3_config['bucket_name']
  config.fog_public = false
  config.fog_authenticated_url_expiration = 1800 # (in seconds) => 30 minutes

  if Rails.env.test?
    config.enable_processing = false
  end
end

s3.yml

test:
  bucket_name: testbucket
  access_key_id: secretkeyid
  secret_access_key: secretaccesskey

test_helper.rb

Fog.mock!
service = Fog::Storage.new({
  :provider                 => 'AWS',
  :aws_access_key_id        => 'secretkeyid',
  :aws_secret_access_key    => 'secretaccesskey'
})
service.directories.create(:key => 'testbucket')

Helpful links:

• Expected(200) <=> Actual(404 Not Found)
• Mocking Fog When Using it with Carrierwave
• How to: Test Fog based uploaders from Carrierwave wiki

Version Info:

• Rails 4.2.6, Ruby 2.2.4
• carrierwave 0.11.0
• fog 1.38.0
• fog-aws 0.9.2

 

Ruby 2.2.2, UTF-8 CSV FILES And Excel

Okay, I’ve taken a bit of a detour with work and find myself supporting a Ruby on Rails application. All brand new to me.

Recently I had to get a CSV file, generated in the Rails app to open in Excel with accents correctly displayed.  The post at Exporting data to CSV and Excel in your Rails apps from plataformatec was very helpful, but for an older version of Ruby.  Here’s what I did to get it working w/Ruby 2.2.2:

	require 'csv'
	module DownloadService
	OPEN_MODE = "w+:UTF-16LE:UTF-8"
	BOM = "\xEF\xBB\xBF" #Byte Order Mark
	def student_list
	File.open("#{file_name}.tsv", OPEN_MODE) do |f|
	csv_file = CSV.generate({:col_sep => "\t"}) do |csv|
	# header row
	csv << ['First_Name', 'Middle_Name', 'Last_Name']
	# put lots of rows in the csv file here
	end
	f.write BOM
	f.write(csv_file)
	end
	end
	end

view raw

csv_for_excel.rb

hosted with ❤ by GitHub

The important point here is that if the file is in UTF-16LE with a BOM, Excel will correctly interpret the file. I found various information about this around the web.  The platformatec posting summarizes the three points that got this working:

1. Use tabulations, not commas.
2. Fields must NOT contain newlines.
3. Use UTF-16 Little Endian to send the file to the user. And include a Little Endian BOM manually.

The open mode indicates the data is UTF-8 encoded but should be written UTF-16LE encoded.

w+:UTF-16LE:UTF-8

Helpful Links:

• Exporting data to CSV and Excel in your Rails apps
• Is it possible to force Excel recognize UTF-8 CSV files automatically? This stackoverflow answer pretty much sums up what I did.  I could not get UTF-8 w/a BOM to work, unfortunately.

Version Info:

• Ruby 2.2.2

Ext JS 4: Ext.ux.RowExpander Tweaks

I recently looked into a few issues we were experiencing with the Ext.ux.RowExpander plugin, which resulted in a few small tweaks to the plugin.

The setup: Grid with checkbox selection model using Ext.ux.RowExpander plugin.  An expanded row is a nested grid.  This was implemented following a pattern that can be found around the web about nesting grids in the expanded rowbody (http://stackoverflow.com/questions/8254113/nested-grid-with-extjs-4).

Issue 1: When the row expands to show the nested grid, the expanded row takes the full width of the grid, instead of aligning to the right of the checkbox column and the expand/collapse column.  This is actually logged as a bug, with pictures at http://www.sencha.com/forum/showthread.php?263231-Content-inside-RowExpander-doesn-t-take-all-width-of-a-row-if-grid-has-CheckboxModel.

The problem is that within the RowExpander plugin, the logic deciding the colspan of the expanded row body does not take into account any other grid features that might impact the columns.  I updated the plugin code to look at the parent grid selection model before determining how wide the colspan should be.

Issue 2: In my UI there is a search panel and then the grid with the search results.  The user can go through and expand various rows, but if the user runs a search again I want the search results to return and the grid refreshed with the results and all the rows collapsed.  What was happening was the grid was refreshed, all rows collapsed, but the expand/collapse icon was in the expanded state.

To fix this I added a method, clearExpanded to the RowExpander plugin.  It’s very straightforward and just clears out the rowsExpanded property the plugin maintains to track which rows are expanded.  Then, I listen for the refresh event on the parent grid’s view, and at that point, call clearExpanded on the plugin.  The refresh event handler on the grid view looks like this:

onMyGridRefresh: function() {
    var rowExpander = this.getSearchGrid().getPlugin('rowexpanderplugin');
    if (rowExpander) {
        rowExpander.clearExpanded();
    }
    this.destroyNestedGrids();
}

Remember, if you are creating nested grids in the manner described above, you need to destroy those grids manually!  Hence, the destroyNestedGrids() method in the code above.  It does a component query on the nested grid xtype and calls destroy on the returned components.

Solution: That’s great, where’s the code?

The updated plugin is at Gist https://gist.github.com/aghuddleston/25b19dfe10f7eb40e54c.   It’s a little long to directly include here.

Version Info: Ext JS 4.1.1

Ext JS 4: Date Display Field

Ext JS Ext.form.field.Display works well for string text, but doesn’t render dates easily. Turns out you can use the method valueToRaw to apply a formatter. Here’s a simple extension for date fields:

	Ext.define('Ext.ux.form.field.DateDisplayField', {
	extend: 'Ext.form.field.Display',
	alias: 'widget.datedisplayfield',
	datePattern: 'Y-m-d',
	valueToRaw: function(value) {
	return Ext.util.Format.date(value, this.datePattern);
	}
	});

view raw

Ext.ux.form.field.DateDisplayField.js

hosted with ❤ by GitHub

This is 100% inspired by this post http://www.sencha.com/forum/showthread.php?148013-How-to-invoke-a-renderer-on-a-displayfield-xtype on the Sencha forums.

To use it’s as simple as having a config on your form like so:

{
  xtype: 'datedisplayfield',
  fieldLabel : 'My Date',
  name : 'myDate'
  datePattern: 'l, F d, Y g:i:s A'
}

datePattern is optional, default it ‘Y-m-d’.

Version Info: Ext JS 4.1.1

Ext JS 4: Simple Splash Page

Sometimes you need a simple splash page while your Ext JS site is loading. Thanks to IvanJouikov at the Sencha forums for the simple solution and helpful post at http://www.sencha.com/forum/showthread.php?230118-Splash-Screen&s=b823569ad51889a57ab5bcef408fc3c6&p=947869&viewfull=1#post947869

I modified a little bit for my situation (no images, just text) and to not use the deprecated center tag.

I added the following to the css:

#loading-parent {
  position: absolute;
  z-index:  20001;
  height: 100%;
  width: 100%;
  padding-top: 50px;
  text-align: center;
}

#loading-indicator {
  font-size: large;
  font-family: tahoma,arial,verdana,sans-serif;
  text-align: center;
  margin: 0 auto;
}

And in the HTML I put this in the body section:

<div id="loading-parent">
  <div id="loading-child">
      <br/><br/>
      <div id="loading-indicator">Loading My Application.  Please wait...</div>
  </div>
</div>

And then the code was just as in the post in the forums. In the launch method of the MVC application:

launch: function() {
  var me = this;
  
  // Create the actual viewport in body
  Ext.create('My.view.Viewport', {
      renderTo: Ext.getBody(),
      listeners: {
          afterrender: function() {
              var mask = Ext.get('loading-mask'),
                  parent = Ext.get('loading-parent');
              // Destroy the masks
              mask.fadeOut({callback: function(){ mask.destroy(); }});
              parent.fadeOut({callback: function(){ parent.destroy(); }});
          }
      }
  });
}

Version Info: Ext JS 4.1.1
Tested on IE8, Chrome and Firefox

Gatling Stress Tool: Example Simulation with a Circular Feed

Here is a simple example of a Gatling Stress Tool simulation that uses a circular feed to populate the request body. I wanted to exercise a dynamic search that I have in the UI by iterating through various different search criteria, thus varying the queries eventually executed by the database. Using a circular feed to populate the body of the request allowed me to do this. The body is JSON and is stored in a tsv file. I had to play with the quotes in the data file a bit to get it to work correctly.

	package mypackage
	import com.excilys.ebi.gatling.core.Predef._
	import com.excilys.ebi.gatling.http.Predef._
	import com.excilys.ebi.gatling.http.Headers.Names._
	import akka.util.duration._
	import bootstrap._
	object FindUsersWithCircularFeed {
	val usersSearchCrit = tsv("usersSearchCrit.tsv").circular
	val scnFindUsers = scenario("Find Users with Circular feed")
	.feed(usersSearchCrit)
	.exec(
	http("Find Users")
	.post("/users/find")
	.body("${crit}")
	.asJSON
	.check(jsonPath("$.success"))
	)
	}

view raw

FindUsersWithCircularFeed.scala

hosted with ❤ by GitHub

We can make this file beautiful and searchable if this error is corrected: No tabs found in this TSV file in line 0.

	crit
	{""searchCrit":[{"searchBy":"USERID","searchText":"hu"}],"page":1,"start":0,"limit":100,"sort":[{"property":"userId","direction":"ASC""}]}
	{""searchCrit":[{"searchBy":"USERID","searchText":"man"}],"page":1,"start":0,"limit":100,"sort":[{"property":"userId","direction":"ASC""}]}
	{""searchCrit":[{"searchBy":"STATUS","status":[7]}],"page":1,"start":0,"limit":100,"sort":[{"property":"status","direction":"ASC""}]}
	{""searchCrit":[{"searchBy":"RESFOR","divisionCode":["HR"]}],"page":1,"start":0,"limit":100,"sort":[{"property":"userId","direction":"ASC""}]}
	{""searchCrit":[{"searchBy":"DIVISION","divisionCode":["HR"]}],"page":1,"start":0,"limit":100,"sort":[{"property":"userId","direction":"ASC""}]}
	{""searchCrit":[{"searchBy":"GROUPID","group":["West_Group"]}],"page":1,"start":0,"limit":100,"sort":[{"property":"userId","direction":"ASC""}]}
	{""searchCrit":[{"searchBy":"LASTACCESSDATE","dateOperator":"ISBEFORE","dateCrit":"2014-01-09T00:00:00-05:00"}],"page":1,"start":0,"limit":100,"sort":[{"property":"lastAccessDate","direction":"ASC""}]}

view raw

usersSearchCrit.tsv

hosted with ❤ by GitHub

Note, this is just the simulation, following the modularization structure in the gatling docs.

Helpful Links:

• Using feeders to construct a StringBody (2.0.0-M3a)
• Circular and random feed does not work as expected

Version Info
Gatling: 1.5.3

Gatling Stress Tool: Utilizing Session User Id in a Simulation

I’ve recently been playing around with Gatling Stress Tool to set up some performance tests. It has proved very easy to get up, running and testing.

For this test case I wanted to run through some CRUD operations and to make sure the object ID was unique, I’m using concatenating the session User Id to the object id. I don’t know scala, so I needed a little help that is readily available from the Gatling group.

Here’s the test case. It doesn’t have the httpConf stuff b/c I’m following the Simulation Modularization structure.

	package mypackage
	import com.excilys.ebi.gatling.core.Predef._
	import com.excilys.ebi.gatling.http.Predef._
	import com.excilys.ebi.gatling.http.Headers.Names._
	import akka.util.duration._
	import bootstrap._
	object Department {
	val departmentScn = scenario("Department CRUD")
	.exec(
	http("Create Department")
	.get("/department/add")
	.queryParam("parameters", (s:Session) => """[{"departmentName":"Perf Test Department","departmentCode":"PerfDept"""+s.userId.toString+"""","initiatorId":"MyIdHere"}]""")
	.check(status.is(200))
	)
	.pause(0 milliseconds, 5 milliseconds)
	.exec (
	http("Update Department")
	.get("/department/update")
	.queryParam("parameters", (s:Session) => """[{"departmentName":"Update name","departmentCode":"PerfDept"""+s.userId.toString+"""","initiatorId":"MyIdHere"}]""")
	.check(status.is(200))
	)
	.pause(0 milliseconds, 5 milliseconds)
	.exec (
	http("Delete Department")
	.get("/department/delete")
	.queryParam("parameters", (s:Session) => """[{"departmentCode":"PerfDept"""+s.userId.toString+"""","initiatorId":"MyIdHere"}]""")
	.check(status.is(200))
	)
	}

view raw

Department.scala

hosted with ❤ by GitHub

And then when it runs I can see the departmentCode is PerfDept1, PerfDept2, etc.

Version Info:
Gatling: 1.5.3

Ext JS 4: Mocking Success and Failure AJAX Responses

Often when building an Ext JS app, I will create a functioning mock-up and work out the design and most client-side issues in my mock-ups. This has a lot to do w/the environment I am working on and it gets down to the fact that it is much faster for me to work with the mock-ups than my real app. Recently I’ve been creating a utility class for my app that has constants and utility methods and one thing that is helpful for me is my “random success/fail” ajax response method. I’ve found this is a great way for me to make sure I’m handling errors everywhere in my app, whether it is through a global error handler or on each AJAX request.

My stripped down utility class looks like this (usually I have a bit more in there, such as constants, app-specific vtypes, and logging.):

Ext.define('MyApp.Util', {
	singleton: true,

	UNKNOWN_ERROR: 'An unknown error occurred',

	randomSuccessFail : function (u) {
		var random = Math.floor(Math.random()*11), url;
		if (random > 8) {
			url = './data/dne.json'
		} else if (random > 3) {
			url = u ? u : '../util/successTrue.json';
		} else {
			url = '../util/successFalse.json';
		}
		return url;
	}

});

“dne.json” is a file that does not exist, so I make sure my mock-ups are handling 404s and other server errors. The other response files look like this:

//successTrue.json
{
	"success":true,
	"message":"success message here, if any"
}

// successFalse.json
{
	"success":false,
	"message":"Failure message goes here and make it a little bit longer and even longer"
}

And it gets called like so in a controller:

// A method inside a controller.  Ext.Ajax request, 
// returning default success/fail response
	saveUser: function() {
		var userDetail = this.getUserDetail(),
			params;

		if (userDetail.getForm().isValid()) {
			this.getDismissAlert().hide();

			params = userDetail.getValues();

			Ext.Ajax.request({
				url: MyApp.Util.randomSuccessFail(),
				jsonData: params,
				scope: this,
				callback: this.saveCallback
			});
		}
	},

// And here's another example, this time on a store load 
// and expecting a different success response
	loadUserIds: function () {
		var combo = this.getUserIdCombo(),
			store = combo.getStore();

		store.proxy.url = MyApp.Util.randomSuccessFail('./data/userids.json');
		store.load({
			scope: this,
			callback: this.loadUserIdsCallback
		});
	},

// Sample callback, shown here... just because
	saveCallback : function (options, success, response) {
		var responseJson, form;
		if (success) {
			responseJson = Ext.decode(response.responseText);
			if (responseJson.success === true) {
				// whatever stuff needs to happen on success
				this.showSuccessResults("Updates saved");
			} else {
				this.showErrorResults(responseJson.message);
			}
		} else {
			this.showErrorResults(MyApp.Util.UNKNOWN_ERROR);
		}
	},

And that’s it, a helpful utility.