Wednesday 15 December 2021

Upgrade to ubuntu 20.04 kills bundle / gem install


The Problem

After upgrading to Ubuntu 20.04, Ruby on Rails gem installations fail. If you use rvm this fails with rvm-managed environments too. Running 

bundle install 
bundle update
gem install

 all fail. 

Error Messages

You will see errors that look like this (in this case it's the byebug gem but other gems will also give the same error):

ERROR: Error installing byebug:

       ERROR: Failed to build gem native extension.

make DESTDIR\= install

make: /usr/bin/mkdir: Command not found

The Solution

The problem is that Ubuntu 20.04 changes the location of the mkdir command, moving it from /usr/bin to /bin. 

The easiest solution is to symbolically link the mkdir command back into /usr/bin.

  1. su - to root (or use sudo to execute the command below)
  2. ln -s /bin/mkdir /usr/bin/

Your bundle should now install as it used to.

FFI gem build failure

Another problem encountered after upgrading to Ubuntu 20.04 is that builds fail referencing the ffi gem. For example, error messags that include this text are seen:
   Ignoring ffi-xxxxx because its extensions are not built

The "xxxxx" part is the ffi version number.

Simply issue this command:
     gem pristine ffi

and the ffi libraries and other artefacts will be reinstated correctly, resolving the problem.

Friday 8 November 2013

Using Apache proxy for Rails / GWT interaction

In my earlier blog about using Rails with GWT, I use Apache's proxy mechanism to bridge the two. So you would put in:
 into your browser and it would route / to your rails server running on port 3000 and it would route another URL (e.g. /MainApplication) to your GWT server running in Eclipse (e.g. http://localhost:8888/MainApplication)

Quite a few people seem to have a problem with the apache proxy config. So here is how I do it on Kubuntu.

I have the standard Kubuntu setup which http2. I also have the proxy module installed (use the Ubuntu/Kubuntu installers for this if you don't have it as a default).

You will find this directory:

In it, create this file: 010-proxy.conf
Note: In the later versions of Apache on Ubuntu you have to have your files end in .conf else they won't be loaded.

And in the file, place YOUR version of this (note that my GWT app is called mainApplication and runs on port 8888 and my Ruby on Rails app runs off root on port 3000):

ServerAlias localhost
ErrorLog /var/log/apache2/
CustomLog /var/log/apache2/ common

ProxyHTMLLogVerbose On
LogLevel Debug

ProxyPass /mainApplication.html http://localhost:8888/mainApplication.html
ProxyPassReverse /mainApplication.html http://localhost:8888/mainApplication.html

ProxyPass /mainApplication.css http://localhost:8888/mainApplication.css
ProxyPassReverse /mainApplication.css http://localhost:8888/mainApplication.css

ProxyPass /MainApplication http://localhost:8888/MainApplication
ProxyPassReverse /MainApplication http://localhost:8888/MainApplication

ProxyPass /sc/ http://localhost:8888/MainApplication/sc/
ProxyPassReverse /sc/ http://localhost:8888/MainApplication/sc/

ProxyPass / http://localhost:3000/
ProxyPassReverse / http://localhost:3000/

DocumentRoot /var/www

Sunday 4 August 2013

Rails4, Phusion Passenger, asset pipeline problems

Rails 4 has introduced many new and exciting features, but it's very new technology and one of the most important features, asset pipelines is causing many people headaches. If you use Phusion Passenger with Rails 4 and Ruby 2, you may encounter these problems:

  1. Pre-compiled fingerprints used for some assets, but not others.
  2. Mismatch between pre-compiled hashes and the hashes that production assigns (i.e. fingerprint mismatches) not resolved by re-compiling the assets.
Usually you will get a message that looks like this:
ActionController::RoutingError (No route matches [GET]
with the asset following in quotes.

We found that the following helped resolve the problems:
  • Append "RAILS_ENV=production RAILS_GROUPS=assets" to your rake assets:precompile line. I.e.: bundle exec rake assets:precompile RAILS_ENV=production RAILS_GROUPS=assets
In config/environments/production.rb:
  • Make Rails use digest: config.assets.digest = true
  • Enable fallback to assets pipeline: config.assets.compile =true
Hopefully in time Rails 4 pipeline issues will be resolved, but for now, these work-arounds in some environments seem to be the best way to ensure that assets remain served in production.

Tuesday 11 June 2013

The TAO of TDD

TDD - you need the right curves in the right places
I have identified a graph that will help maximise the benefits of test driven development. This takes into account the natural phases that a software project typically goes through as well as a often ignored but very significant bit of code meta data: Code volatility. Because, implicit in the notion of TDD is the idea of code volatility.

A closer look at the bandwagon
TDD (test driven development) is all the rage right now. It means that you write your tests first, then you develop your code, checking it against the tests.

The idea is that by knowing how you will test your code, you gain not only from having a robust set of tests written to prove that your code is correct, but by developing the tests in the first place, you have focussed on the finer details of how that code will work. For enterprise class development, this means that your development process is something like this:

  1. System requirements
  2. Problem analysis/solution identification
  3. Program specification
  4. Develop tests
  5. Write code running tests for each section of code as you go
  6. Verify code against full test suit
  7. Higher level human testing
The befits are:
  • Better clarify of code before start of programming
  • Full test suit available for verifying subsequent code changes/maintenance.
  • Focussing on, and agreeing tests up front is an excellent way to debug requirements and identify poorly specified, misunderstood or ambiguous requirements.
  • Rapid testing: Automated testing is usually far quicker than human testing.
  • Full set of automated, repeatable tests ensures all tests reliably performed and at low cost.
  • By running comprehensive tests across the entire software suite, unexpected/unintended consequences of code changes/development can be identified that could otherwise be missed in the more "targetted" approach of human testing.
  • Human testing can be focussed on higher value, more intelligent, expert level testing. Better job satisfaction for humans.
  • By removing the cost of "mundane" testing, human expertise is freed to identify nuances such as user experience issues, colour/icon mismatches, porting problems and other such "higher level" verifications.
  • With human testing focussed on the "interesting" work, a higher quality testing results. Bored humans make for poor testers.
  • The biggest advantage of all is that quite simply, much more testing can be performed much more regularly and much more comprehensively.
A bit of magic
However, there is a bit of a black art to TDD. One that is often overlooked: TDD is about delivering benefits in terms of cost and time.

TDD implies that you are writing tests, lots of them. This means:
  • Your tests need to cover a potentially exponential set of execution paths, combinations, boundaries and exceptions, component dependences, flooding, stress testing, performance testing and data set-up.
  • This implies a significant amount of test scripting. And that means software development. Often to fully test an item of code requires more lines of code than the code being tested.
  • Test scripts, like all software, are prone to error. Bugs in test scripts can waste significant time if they falsely identify code bugs. You start by attempting to resolve a problem that doesn't exist. Depending on the nature of the test, the number of dependencies in the code and the complexity of the code, this can be a significant misdirection and hence waste of time and resources.
  • A large package of test scripts and data becomes a significant factor when changing code. This is not limited to software maintenance. Software additions almost always require existing code to be modified in some way and that usually implies updating the testing software and data too. In software with high combinatorial paths or outcomes, this can imply very serious test script modification overheads.
TDD is about benefits. These benefits include cost savings, software reliability, reduced cost of maintenance and faster delivery of code changes. One word is at the heart of these benefits: Automation.

However, automation implies repetition in this context. Scripting a test that will only ever be run once is unlikely to yield a cost-benefit in your favour. The downside of additional software development (test scripts/data), potential for bugs in the test scripts and other considerations mentioned above need to be looked at carefully as traditional human testing may well be better and possibly cheaper in these circumstances.

Likewise highly volatile software undergoing major, frequent changes may amplify the additional software maintenance costs of the test suite past usefulness. Code/functional volatility essentially draws software closer to the curve delineating that software as "once-off" code and hence much less likely to yield a scripted TDD benefit.

When human is best: Meet the curve
Note that TDD as a methodology doesn't require scripting. It's perfectly feasible to implement your testing as human testing although TDD as it's commonly used implies scripted testing.

When the software volatility curve is such that it draws close to once-off code, then human testing begins to beat scripted testing.

A "typical" software lifetime follows a volatility curve, with high volatility in the early stages, slowly reducing over time. If you include the conceptualisation and requirements phase in that graph, then the volatility curve is even more dramatic as various ideas, scoping decisions and requirements inclusions/exclusions take place.

To begin writing your tests too early in the software cycle, during the highly volatile stages, is to dramatically increase the costs and also the likelihood of redundant or changing tests.

Writing your tests too late in the software's life, for example several generations into the software when it's stable, has few defects and its requirements are unlikely to change means that you are developing tests to prove already proven software. The majority of these tests are unlikely to be reused much,if at all, due to low software maintenance. The majority of the test suite would therefore be redundant.

Therefore, there is a "sweet spot" in most software developments where scripted TDD is appropriate. It's between a certain point at the start of the software's life-cycle and ends at a certain point of software stability and maturity. Within this sweet spot the return on investment of scripted TDD is positive and worth doing. However, outside of this sweet spot, the return on investment is at best close to zero, but often it is significantly negative and hence scripted TDD is not worth investing in when these conditions hold.

This implies that there is a natural value curve to TDD across a project's life. That curve begins and ends either negatively or close to zero with a positive "sweet spot" yielding good return on investment for TDD.

Identifying this curve enables project planning that will yield the most benefit from TDD with well timed allocation of resources maximising the returns on investment.

Tuesday 9 April 2013

No identification of primary key on legacy database table

When using Rails against existing databases, it is unlikely that the Rails naming conventions for tables has been used. This means that models are likely to require the table name to be specified.

The old directive set_table_name has been removed from Rails. You now need to use:
self.table_name= :theTableName
However, this appears to have introduced a break, at least in Rails 4. Specifying the table name on a table that has id as the key column doesn't result in Rails identifying the key column, you still need to specify it:
self.primary_key= :id
 This means that whenever you set the table_name, you have to also set the primary key column name.

Monday 1 April 2013

Part II - Creating beautiful reports in Ruby on Rails

In Part 1, we looked at the principles behind creating beautiful reports in Ruby on Rails. Please read part 1 (see below) before reading this posting.


You now need to install LibreOffice.See Part I for why we are using LibreOffice. You can get the download as well as installation instructions here.

At this stage, you should really install LibreOffice on your desktop/laptop as well because you will be using it to create the template.

The LibreOffice template will contain placeholders that the Ruby code will then substitute at runtime for real values. These place holders are text values within the document and table column rows.

Creating the template
The beauty of this method of creating reports is that you can use LibreOffice to design the reports.
  1. Start up LibreOffice.
  2. Click "Text Document". This should open a new document. Note that with LibreOffice you can create template files and set the default template as we have done for our SMS Speedway mobile messaging service. This is handy if you are going to be creating many reports that should include standard features like company logo, headers, footers, etc.
For the purposes of this document, we will create a simple report. We will base it on our excellent business text messaging service, SMS Speedway, to report on text messages that were unable to be delivered (e.g. incorrect mobile number). Having written the report, we would be able to use the same procedure described here to expand the report to include statuses of other services that form a part of SMS Speedway, for example social media communications that were unsuccessful (e.g. timed out due to recipient not logged into FaceBook, Twitter, or Google Plus).

Expanding the report would be a simple case of creating a subsequent section in LibreOffice and putting in the appropriate fields as described here. Right now, we have a blank document, so let's get started.

This is what our template looks like:
Exception report template screenshot

The logo, title (SMS EXCEPTIONS REPORT) and the created field are in the header section of the document. Click this link if you would like help on how to create headers and footers in LibreOffice. What's important to note is our first bit of templating that will later be populated by our Ruby code:

Screenshot showing a marker for fixed text

[UPDATE to section below] Please note: The creator of this gem has written to me and asked that I point out that the use of fields is now redundant and won't work with certain versions of LibreOffice. You should merely use the [XXX] placeholders as plain text. I have left in this (old) method of achieving this as as some people may still find it useful for certain gem/libreoffice  combinations. But please use plain text markers first!

The entry "[TODAYS_DATE]" will be replaced by our ruby code with the current date every time the report is run. This is entered as a user field.
  1. Place the cursor at the point where you wish to enter the placeholder that will later be substituted for the real value. In this case, it's where we see [TODAYS_DATE] in the image above.
  2. Insert/fields/other (or ctrl-F2).
  3. In the Value field, it is important to enter the placeholder name IN CAPS and encased in square brackets ("[" and "]").
  4. Click the green tick and then the "insert" to insert the field.
Entering a user field
You should now see [TODAYS_DATE] as shown in the previous image.

Later, when we pass this template to the Ruby code, it will look for the text value TODAYS_DATE and it will substitute the current timestamp. An example is shown here:

Text marker with value substituted

Usually, with a report, you need to be able to group data items together. In our example, we have transmission files, meaning the files that clients can upload to the SMS Speedway server that contain messages to be sent (if they do not want to use the web site or integrate their software using our extensive API).

If a client uploads more than one transmission file, for each file, we wish to report the exceptions. This therefore forms a logical grouping, to be repeated for each file in the report.

To enable this grouping, we need to create LibreOffice sections. In our example, our section is called MAIN_SECTION. Note that there are no enclosing square brackets ("[" and "]") and that the name is in CAPs. This name (shown below) can be seen at the bottom of the document window when you have clicked into the section.
Showing the section name in LibreOffice

To insert a section, choose Insert / Section:
Inserting a section in LibreOffice

 Subsequent items such as the transmission file name and the table containing the exceptions data can now be entered in the section. This will enable the report to duplicate all the items in the section for each transmission file.

Free text placeholders
Where less precise formatting is required, you can simply add the placemarker, enclosed in square brackets ( "[" and "]") in the text. And example in our template above is this bit:
Plain text placeholder

In this case, the [USER_REF] is simply plain text as we have the rest of the line as space and no special formatting requirements,

Perhaps one of the most useful layout options is the use of tables. In our template, we have:

However, the table needs to be named. As we can see here at the bottom of the Document window:
Named table

  1. Insert the table (Insert/table)
  2. Create your columns, if you want to have headings, enter the headings i the first row.
  3. Set the table name: Table/Table properties then click the tab Table.
  4. Enter a name, it must be in CAPS, but as opposed to text and user field value strings, the table name MUST NOT be enclosed in square brackets:
Table name
Now you need to enter the column value placeholders. These, as per text and user field placeholders must be CAPS and also enclosed in square brackets ("[" and "]"). These can be seen in the image of the table above ([MOBILE], [STATUS] and [MESSAGE]).

It is important to note that some items need to be encased in square brackets and others not. All placeholders must be CAPs. In general, placeholders need to be encased in square brackets but object names (section, table) do not have square brackets.

Saving the file
Save your file (file/save). You should now have your template file of type .odt.

The Ruby code
Now you need to write the Ruby code. The template handling magic is implemented in the gem odf-report. Our SMS Speedway system is based on the enterprise class Postgres DBMS so we include the Postgres gem. We also make provision for emailing reports, hence the mail gem:
require "rubygems"
require "pg"
require "odf-report"
require 'mail'
Data Classes
We read the data from the database, but we store each data item in an object such that the odf-report gem can access it and obtain the data.

For example, the report has several transmission files. The template (see above) has the [USER_REF] placeholder for the transmission file. So we define the class as follows:

class TransmitFile
  attr_accessor :user_ref, :timestamp, :msgs
  def initialize(_user_ref, _timestamp, _msgs=[])
Note the attr_accessor which is effectively Ruby shorthand to define the setters and getters for the variables in the class by the same name.

From the above, you can see that @msgs is an array. I.e. Each file is associated with some messages. To define the structure to hold those messages, we have:
class Msg
  attr_accessor :number, :status, :message
  def initialize(_number, _status, _message)
 And this means that for each message, where @currTransmitFile holds the current transmit file, we simply do this:
@currTransmitFile.msgs <<,status,msg)
In this manner, we can select from the database, all the transmit files that need to be in the report as an outer loop with an inner loop selecting all the relevant messages for the current transmit file and associated as shown above. Each file, once its messages have been associated with it, is then added to the @items array before the next file in the outer loop is processed:
        @items << @currTransmitFile
 We therefore end up with @items containing a list of all the transmit files that should be in the report, with each file having all the messages that should be in the report for that file associated with is in the msgs attribute.

Let's quickly recap:
  • We have a .odt file, created in LibreOffice that has a template.
  • The template has a report timestamp field called TODAYS_DATE
  • It has a section called MAIN_SECTION
  • Within that section there is a free text placeholder called USER_REF
  • and there is also a table called EXCEPTION_TABLE that has three colums with placeholders in each: MOBILE, STATUS and MESSAGE.
  • We also have an array of data objects of type TransmitFile with the user_ref field in it.
  • Each TransmitFile has an array of objects of type Msg that have the fields number, status and message that map to the columns in the template's EXCEPTION_TABLE table field.
Merging the data into the template
Now we need do a data merge on the template. First, we need to instantiate the ODFReport module. We have called our template exceptionsReport_template.odt:
report ="exceptionsReport_template.odt") { |r|
 The above creates our report object and associates the template file with our report.

Now we need to tell ODFReport about the placeholders, tables and sections in our template:

report ="exceptionsReport_template.odt") { |r|
  r.add_field :todays_date,'%Y-%m-%d %H:%M:%S')

  r.add_section(:MAIN_SECTION, @items) do |s|

        s.add_table("EXCEPTION_TABLE", :msgs, :header=>false) do |t|
           t.add_column(:mobile, "number")
           t.add_column(:status, "status")
           t.add_column(:message, "message")

Note that the placeholders in our template are associated with Ruby symbols of the same name, but take careful note of the casing, it's important!
  •  :todays_date is associated with [TODAYS_DATE], likewise :user_ref.
  • :MAIN_SECTION is associated with MAIN_SECTION
  • however the column placeholders are again lowercase that maps to uppercase placeholders in the template ( mobile, :status, :message).
The r.add_section call associates the section in the template called MAIN_SECTION with the object array @items. ODFReport will iterate through each object, creating a copy of MAIN_SECTION for every iteration, and filling in the placeholders each time if they have been associated with attributes in @items.

This is how those associations are made:

The s.add_field call associates template placeholders with values. The first parameter is the template placeholder name as described above, the second is the value.

With :todays_date, we  pass through the current time formatted exactly as it should appear. This is a fixed value.

However, for the :user_ref, we pass :user_ref as the second parameter. this is telling add_field that the attribute called user_ref in the @items item under consideration, contains the value for this placeholder.

Likewise, when we tell ODFReport about the table called EXCEPTION_TABLE in the report (add_table), the first parameter gives the table's name in the template (EXCEPTION_TABLE), the second parameter tells it that the array called msgs in the item under consideration contains the table data objects.

As with add_field, the add_column method associates a template table column placeholder with a data object attribute. Hence:
           t.add_column(:mobile, "number")

tells the ODFReport module that placholder [MOBILE] in the template is associated with the object attribute "number" in the current msgs object.

ODFReport will iterate through all the msgs, object by object, and fill in each row, be getting the attribute for each column's placeholder.

Generating the report
Now that we have associated the template placeholders with the data structure that we populated from the database, all that remains is to generate the report.

We want the report to go to a file called exceptionsReport.odt. The following call will create the report:


However, we want this to be a cross-platform report, readable on all devices and on all operating systems without the need for office software to be installed. The best file format is PDF, so we need to convert the .odt file into a .pdf file. This is where the magic of headless LibreOffice comes into its own:
`/usr/bin/libreoffice4.0 --headless --invisible --convert-to pdf ./exceptionsReport.odt`
The ` in Ruby executes the command as a sub-process. The result is a file called exceptionsReport.pdf.

And there you have it, your beautiful report, created as a PDF!

For your convenience, here is the working part of this code:
report ="exceptionsReport_template.odt") { |r|
  r.add_field :todays_date,'%Y-%m-%d %H:%M:%S')

  r.add_section(:MAIN_SECTION, @items) do |s|

        s.add_table("EXCEPTION_TABLE", :msgs, :header=>false) do |t|
           t.add_column(:mobile, "number")
           t.add_column(:status, "status")
           t.add_column(:message, "message")

report_file_name = report.generate('./exceptionsReport.odt')
`/usr/bin/libreoffice4.0 --headless --invisible --convert-to pdf ./exceptionsReport.odt`
Design without the pain
Now it is possible for your graphic designers or the marketing department to redesign the reports. Perhaps they want to market a promotion or change the font. Maybe they need to update the company contact details or change the logo.

As long as they keep the placeholder names, section names and table names the same, they can do whatever they like. It's merely a matter of replacing the template file with the new one and the report is instantly updated without any code changes required.

Customised reports, common code
Another advantage to this technique is that by specifying your template, you can create highly customised reports (e.g. with a client's logo on it) yet keep your code common.

Through careful planning, a wealth of beautiful, tailored output is possible without the headache of code impact, testing cycles and release procedures and other hoops normally associated with application releases, no matter how minor they are.

A major hole in Ruby and Rails' enterprise class capabilities has been filled. A large section of business software is all about the output from that software and entire teams in enterprises are devoted to just that task.

Ruby and Ruby on Rails are now enabled to play a role in serious enterprise and business information and reporting software. A new era has begun.