Transferring ownership of repositories on GitHub

For the past couple months, I’ve been studying. As a side effect, my GitHub account was cluttered with code that is experimental. I didn’t exactly want to trash the experimental code. I wanted to keep my code but also not specifically keep it under my profile.

The solution I found was to create an organization and transfer my desired repositories there. I thought this was a great solution, there was just one little detail I was missing: the current GitHub API does not support repository ownership transfer. Which meant I would have to go to each repository, click on “Settings”, click on “Transfer”, fill in the “Repository Name” and then put the “Destination User”. A lot of steps for someone looking to move over 250 repositories.

The first thing that came to my mind was to use Selenium to automate this task. But my lack of exposure with the technology made me think a bit outside the box. One of the things I learned these past months was capybara. Capybara is used alongside Rspec, extending the test suite DSL. It mimics user interaction with the browser and comes with a Selenium driver out of the box.

In other words, you can create a bot to go to the browser, fill up forms and submit it. This is exactly what I was looking for. As I stated before, Capybara uses Rspec, so my code would actually have to be wrapped inside a test.

Caveats

You need to disable two-factor authentication on GitHub, and after the script finishes running set it back on.
It does not transfer private repositories.
You need Mozilla’s geckodriver installed. If you use macOS, you can use brew to install it.

The Script

This was developed as a hack. Use at your own risk. You can download it at gabidavila/github-move-repositories. As of now this code gets all public repositories, unless the variable ONLY_FORKS is set to TRUE, and moves to a destination user. Do not push your .env file.If you want to move only specific repositories, you will need to edit the code yourself. For more information the README.md of this project is kept up to date.

Contributions are welcome if you feel you can help improve the tool. For example: add options of which repositories to move.

Enjoy!

Using Active Record migrations beyond SQLite

SQLite is really a good tool to set up quick proof of concepts and small applications; however it’s not the most robust solution on the market for working with relational databases. In the open source community two databases take the top of the list: PostgreSQL and MySQL.

I did a small project for my studies. I was using SQLite as I didn’t need much out of it. Curious, I decided to see how the application would behave on other databases and decided to try PostgreSQL and MySQL. I had two problems to solve, and this post is about the first one: how to deal with the migrations. They were as follows:

	class CreateArtists < ActiveRecord::Migration
	def change
	create_table :artists do \|t\|
	t.string :name

	t.timestamps
	end

	add_index :artists, :name
	end
	end

	class CreateSongs < ActiveRecord::Migration
	def change
	create_table :songs do \|t\|
	t.string :title
	t.integer :artist_id

	t.timestamps
	end

	add_index :songs, :title
	add_foreign_key :songs, :artists
	end
	end

view raw ar_migration_snippet1.rb hosted with ❤ by GitHub

Active Record automatically put the field id in all of its tables, that’s why it is omitted on the migrations.

In PostgreSQL it went smoothly, all the migrations ran without any hiccup, except on MySQL, it gave me an error!

StandardError: An error has occurred, all later migrations canceled:

Column `artist_id` on table `songs` has a type of `int(11)`.
This does not match column `id` on `artists`, which has type `bigint(20)`.
To resolve this issue, change the type of the `artist_id` column on `songs` to be :integer. (For example `t.integer artist_id`).

Original message: Mysql2::Error: Cannot add foreign key constraint: ALTER TABLE `songs` ADD CONSTRAINT `fk_rails_5ce8fd4cc7`
FOREIGN KEY (`artist_id`)
REFERENCES `artists` (`id`)

The problem, beyond generating an ineligible name for an index: fk_rails_5ce8fd4cc7, is that artist_id on my table was as INT. The first thing I checked was to see if the artist.id was UNSIGNED and if my foreign key was also unsigned. They weren’t, but since were both signed, it wouldn’t throw an error. Looking more closely to the error message I noticed that the type in my foreign key column did not match the type on the primary key on the other table. Little did I know that Active Record generates the id field not as an INT, but as BIGINT.

I decided to go back and look at PostgreSQL, and to my surprise, and up to now I still am not sure of why, PostgreSQL did allow the column type mismatch where MySQL threw an error.

To fix it, I had to change the migration as follows:

	class CreateSongs < ActiveRecord::Migration
	def change
	create_table :songs do \|t\|
	t.string :title
	t.integer :artist_id, limit: 8

	t.timestamps
	end

	add_index :songs, :title
	add_foreign_key :songs, :artists
	end
	end

view raw ar_migration_snippet2.rb hosted with ❤ by GitHub

Digging online, I found out how to create a bigint field with AR. According to the post, this would only work on MySQL, which they did, but I found it also worked with PostgreSQL (I tested MySQL 5.7 and Postgres 9.6): t.integer :artist_id, limit: 8.

The limit is used to set a maximum length for string types or number of bytes for numbers.

Why type matching is important

As an INT let’s say you can fit your number inside an espresso cup. Sure you can use the Starbucks Venti size cup to fit your coffee, but the full content of a Venti would never fit an espresso cup.

In the specific domain I am working on if I had a big list of Artists, and happen to have an artist which ID was higher than 2,147,483,647 (signed, and for both PostgreSQL and MySQL), I would get an error when trying to insert it into the Songs table since an Artist id can be up to 8 bytes (9,223,372,036,854,775,807).

Example:

Queen has its Artist id as: 21474836481 (which is a BIGINT)

Trying to insert “We Will Rock you” in the artist_id column for songs:

	INSERT INTO songs (title, artist_id, created_at, updated_at)
	VALUES ('We will Rock you', 21474836481, now(), now());

view raw ar_migration_snippet3.sql hosted with ❤ by GitHub

We get:

********** Error **********

ERROR: integer out of range
SQL state: 22003

This is the kind of problem we don’t usually notice in the beginning, and more often than not while the application is in production for even years, but this can happen and will happen if we don’t pay attention to foreign key types.

After that change, all the migrations ran smoothly. And I could actually move forward to the next problem (and post): Filtering a song title or artist name.

ActiveRecord: Has Many Through Through Relationship

Developers in general love when stuff works. Having a solution that can solve about 80% of your problems can leave time for you to deal with the other 20%.

But this post is not about Active Record vs. Data Mapper or any thing like it. Each one has its use case where it’s best applicable and it depends on you (or your team) to decide which to use. Keep in mind that with Active Record (AR), domain concerns and persistence concerns are mixed together and that with Data Mapper (DM), domain concerns and persistence concerns are kept separate.

Let’s talk about magic. How magical AR can be and how it can make your life easier. The beauty of programming is that two different individuals can reach the same result using different routes even if using the same tools. The convention over configuration that some frameworks like Laravel and Rails use makes everything feel so effortless, while actually under the hood, there is a lot going on.

Solving a code challenge

This week I was given the following schema on this code challenge:

Database Mapping

With four models:

Boat:
- belongs to a Captain
- has many records of BoatClassification
- has many records of Classification throughBoatClassification
Captain
- has many records of Boat
BoatClassification
- belongs to a Boat
- belongs to a Classification
Classification
- has many records of BoatClassification
- has many records of Boat through BoatClassification

And here is the code in Ruby:

	### app/models/boat.rb ###
	class Boat < ActiveRecord::Base
	belongs_to :captain
	has_many :boat_classifications
	has_many :classifications, through: :boat_classifications
	end

	### app/models/captain.rb ###
	class Captain < ActiveRecord::Base
	has_many :boats
	end

	### app/models/boat_classification.rb ###
	class BoatClassification < ActiveRecord::Base
	belongs_to :boat
	belongs_to :classification
	end

	### app/models/classification.rb ###
	class Classification < ActiveRecord::Base
	has_many :boat_classifications
	has_many :boats, through: :boat_classifications
	end

view raw models.rb hosted with ❤ by GitHub

The models were given to me as shown above, including the relationships. Stuff started easy, like:

Class: Boat -> Retrieve all boats without a Captain:

Boat.where(captain_id: nil) which translates to:

	SELECT `boats`.*
	FROM `boats`
	WHERE `boats`.`captain_id` IS NULL

view raw ar_snippet_sql1.sql hosted with ❤ by GitHub

But then, stuff started to get a bit more complicated…

Class: Boat -> Retrieve all boats with three Classifications:

My thought: This one I got it! The code already showed me the has many through from Boat to Classification, now what I need to do is GROUP BY boats.id and all will be fine…

Boat.joins(:classifications).group("boats.id").having("count(classifications.id) = ?", 3)  

view raw ar_snippet_ruby_1.rb hosted with ❤ by GitHub

Active Record saved me from a lot of trouble from doing the following query:

	SELECT `boats`.*
	FROM `boats`
	INNER JOIN `boat_classifications`
	ON `boat_classifications`.`boat_id` = `boats`.`id`
	INNER JOIN `classifications`
	ON `classifications`.`id` = `boat_classifications`.`classification_id`
	GROUP BY `boats`.`id`
	HAVING count(classifications.id) = 3

view raw ar_snippet_sql_2.sql hosted with ❤ by GitHub

This are 8 lines of code translated into one!

magic trick

Ok, I thought, we are going places with this. Give me one more!

Class: Captain -> Retrieve all Captains that pilot a specific Classification of a Boat

I knew the SQL code for this one! It’s easy when coding to end up doing chained queries with subqueries inside. I wanted to avoid that as much as possible knowing that would be able to solve the question with joins.

Reading the documentation I saw that Ruby’s Active Record gives us tools to avoid this kind of situation, one of them are the relationships. It’s possible to say looking at the classes and diagrams that Classification and Captain have a nested relationship. A has many through through if you like, yes that’s “through” twice.

Diagram

One way to remember is to look at the model Captain and Boat:

	class Captain < ActiveRecord::Base
	has_many :boats
	end

	class Boat < ActiveRecord::Base
	belongs_to :captain
	has_many :boat_classifications
	has_many :classifications, through: :boat_classifications
	end

view raw ar_snippet_ruby_2.rb hosted with ❤ by GitHub

:boats is a relationship for Boat. This means I can do a join, (specifically a nested one):

Captain -> Boat -> BoatClassification -> Classification

Captain.joins(boats: {boat_classifications: :classification})

view raw ar_snippet_ruby_3.rb hosted with ❤ by GitHub

confused, oh wait!

Ok, that makes sense, through :boats I have access to :boat_classifications which in turn has access to the :classification relationship. But, :boats also has access to :classifications, making this possible:

Captain -> Boat ->> Classification

Captain.joins(boats: :classifications)

view raw ar_snippet_ruby_4.rb hosted with ❤ by GitHub

And we finally add the filter to the query:

Captain.joins(boats: :classifications).where(classifications: {name: 'Sailboat'})

view raw ar_snippet_ruby_5.rb hosted with ❤ by GitHub

Saving us from having to write this:

	SELECT
	`captains`.*
	FROM
	`captains`
	INNER JOIN
	`boats` ON `boats`.`captain_id` = `captains`.`id`
	INNER JOIN
	`boat_classifications` ON `boat_classifications`.`boat_id` = `boats`.`id`
	INNER JOIN
	`classifications` ON `classifications`.`id` = `boat_classifications`.`classification_id`
	WHERE
	`classifications`.`name` = 'Sailboat'

view raw ar_snippet_sql_3.sql hosted with ❤ by GitHub

TL;DR;

Chaining method calls on the model’s class always returns the model itself
Hashes are more used than you would imagine
Avoid subqueries
Magic happens through relationships (which saves you from the subqueries)
Putting things on diagrams is not a question of being fancy, but rather to be able to better visualize problems.

Bonus – Performance

If we had used the subquery for searching on the last category we would have:

	SELECT
	`captains`.*
	FROM
	`captains`
	WHERE
	`captains`.`id` IN (SELECT
	`boats`.`captain_id`
	FROM
	`boats`
	WHERE
	`boats`.`id` IN (SELECT
	`boat_classifications`.`boat_id`
	FROM
	`boat_classifications`
	WHERE
	`boat_classifications`.`classification_id` IN (SELECT
	`classifications`.`id`
	FROM
	`classifications`
	WHERE
	`classifications`.`name` = 'Sailboat')));

view raw ar_snippet_sql_4.sql hosted with ❤ by GitHub

Doing a query cost analysis on it, by adding an EXPLAIN in the beginning of the query, it returns:

SubqueriesAR

But using the correct relationships we have:

Don’t worry much about the numbers, but look more at the colors, by using the existing foreign keys we avoid doing a full table scan on the tables, even with a join of four tables the query plan showed that subqueries are 30% more slower than using the existing indexes and relationships.

Playing with Ruby modules: prepend

In the past month or so I’ve been learning Ruby, this is my first blog post about the language and my experiences learning it. I’d love to hear your feedback.

Ruby allows you to work with modules and mixins, giving characteristics to classes and instances that otherwise could only happen through inheritance. One can argue that there are many similarities between Ruby’s mixins and PHP’s traits.

But first things first: what is a mixin? A mixin is a module that you include in a class, because you are “mixing in” the module methods with the instance methods of a class.

When learning about Ruby’s modules behavior, specifically the include functionality, I couldn’t help but think how similar it was to PHP traits. It reminded me how a class can override an inherited method from a trait, or just plain inheritance of new features that class didn’t have before.

What I learned was the following:

classes that extend a module have methods from it, however they are only accessible through a class call
- extend doesn’t change the inheritance chain
classes that include a module have methods from it, however they are only accessible through an instance call
- include does change the inheritance chain
and, finally; prepend. Which is a funny case in my opinion, a bit trickier to grasp the concept at first sight. But I got a bit more understanding after playing with the following. I think…

Prepend

I learned about the existence of prepend after doing this small exercise. Before this I thought you either extend a module or include it.

Try out the following code:

What should we get when calling #color?

car = Car.new
car.color

The naive me, thought in the beginning the result would be:

# Red
# =&gt; nil

And my reasoning was, except for monkey patching, isn’t Car overriding everything because when instantiated we overload the method #color? To my surprise, it printed (come on, go check it!):

# Blue
# =&gt; nil

Apparently I was wrong. Playing with pry I added super (LOC4) to Vehicle#color, convinced that the return would be Green, since Car inherits Vehicle, right? Wrong again!

	module Paintable
	def color
	puts "Blue"
	super
	end
	end

	class Vehicle
	def color
	puts "Green"
	end
	end

	class Car < Vehicle
	prepend Paintable

	def color
	puts "Red"
	end
	end

	car = Car.new
	car.color
	# Blue
	# Red
	# => nil

view raw prepend-02.rb hosted with ❤ by GitHub

It looks like, Paintable is overriding Car, which is a bit confusing because the keyword is prepend, for all I know (and the dictionary too, trust me, I checked) it means to attach something in the beginning of something else.

With that in mind, shouldn’t Paintable come before Car, and because I have overridden #color in Car (LOC17), shouldn’t Car have the last say in what color should be?

Apparently, no. The correct way to read it is:

Paintable is prepending the inheritance chain of Car

And this is why:

	# without the prepend, extend or include
	Car.ancestors
	# => [Car, Vehicle, Object, PP::ObjectMixin, Kernel, BasicObject]

	# with the extend
	Car.ancestors
	# => [Car, Vehicle, Object, PP::ObjectMixin, Kernel, BasicObject]

	# with the include
	Car.ancestors
	# => [Car, Paintable, Vehicle, Object, PP::ObjectMixin, Kernel, BasicObject]

	# with the prepend
	Car.ancestors
	# => [Paintable, Car, Vehicle, Object, PP::ObjectMixin, Kernel, BasicObject]

view raw prepend-03.rb hosted with ❤ by GitHub

Which means that when calling #color in the instance we get Blue and why super prints Red, since Printable now has Car as a superclass.

Conclusion and Use Cases

Modules add functionality to your class, either by being a mixin or through class methods when you extend it.

I found this blog post with a good example of inheritance vs mixins:

Inheritance means that a class is a “type of something” and suggests specialisation. For example, a Pikachu object is a type of Pokemon and so it makes sense to inherit from the Pokemon class.

When a class should be capable of something, you should use a Mixin. For example, DVD, MP3, and Bluray classes all have the #play method, but just because they are all capable of the same action, does not mean they all should inherit from the same parent

Keeping the Car example, the Paintable module makes more sense to be used with an extend, since you can change colors not only of cars, vehicles, multiple types of objects that won’t necessarily inherit Vehicle.

Also, beyond including, and extending a module you can prepend it. prepend changes the inheritance chain as does include, but with inverse order. extend does not change the inheritance chain.

You will be want to use prepend whenever you want your code to be executed before the class. After working with so many legacy applications, I can see this being really useful when you want to extend the functionality of some class, let’s say Login, you want to run before Login a check to see if the requester address is accessing only through your closed network, prepend would allow you to do that. And a good side effect of using prepend is that you remove the necessity of directly monkey patch a method.

I think only experience will make clear the best way to implement a certain module in an application, but understanding how each of these tools behave is important before you can make that decision.

From MySQL 8.0.0 to MySQL 8.0.1 – or any other dev milestone

Disclaimer: This post is aimed to you, the curious developer, sys-admin, technologist, whatever-title-you-use. DO NOT run the following lines on production. Not even in a stable environment, do this if you don’t care about the outcome of the current data.

If you want to keep up with the newest MySQL developer milestones I have news for you: there is no upgrade available for milestone versions. The way to go is to remove old version and install new one, according to their website:

Upgrades between milestone releases (or from a milestone release to a GA release) are not supported. For example, upgrading from 8.0.0 to 8.0.1 is not supported, as neither are GA status releases.

So if you, like me, had the 8.0.0 version and want to test the 8.0.1 (alhtough 8.0.3 milestone is already in development) you need to do something like the following (tutorial based on Debian/Ubuntu servers).

Stop your service:

$ sudo service mysql stop

Download Oracle’s repository and install it, as of now this is the current version, you can get the new package here:

$ wget https://dev.mysql.com/get/mysql-apt-config_0.8.6-1_all.deb
$ sudo dpkg -i mysql-apt-config_0.8.6-1_all.deb

Clean your old install, you will lose all the data. Be careful, back up is on you!

$ sudo apt-get remove --purge mysql-server mysql-client mysql-common
$ sudo apt autoremove
$ sudo apt-get autoclean
$ sudo apt-get install mysql-server

This is the way to go to test the new features such as Descending Indexes and others. Remember, the new default encoding was changed from latin1 to utf8mb4.

Short feature list:

Descending Indexes
New Data Dictionary
New GROUPING() function to distinguish NULL values for super-aggregated rows, more information
Recursive queries using WITH

The complete list is available here.