When my plan to push my updates to GitHub using simpleGit didn't go as planned, I was certainly expecting to write "but when pushing over the GitHub API the updates worked as expected". But they didn't. I was having the same problems over the API as when using simpleGit locally.

The problem, I think, is the same as before, and is similar to the problems with fs.writeFile that Dave described last week. The server is synchronous and will call several publish callbacks at the same time. To reliably push to GitHub, I need to deal with the updated files asynchronously, one after the other. And since the publish callbacks run inside the server, I don't really want to do asynchronous things inside my publish callbacks, since I could then slow down or even block the server.

My idea now is just to save the updates someplace as json, then write a script running separately from nodeStorage to handle those updates one at a time. I'll do my best to give the json files unique filenames, using a timestamp with Date.now() and the md5 checksum of the body of the file being written. Here is my script.

The next step will be a separate script to process those json files one at a time as they are produced. 


This didn't actually turn out the way I planned when I was setting everything up for GitHub

I thought that using git from a callback would be just like using it when I'm using it to save code. Do 'git status' to see what needs to be done, 'git add' to set up what to commit, 'git commit' to save to the local repository, and finally 'git push' to send the changes to GitHub. 

Since the simple-git module is included with nodeStorage, it can be simply loaded into our callback. In theory it would just be

var ghRoot = "/home/shanson/test-rhino"; //where I cloned the repository
var simpleGit = require('simple-git')(ghRoot);
filesystem.newObject(ghRoot + '/' + relpath, body);
simpleGit.status().add('.').commit('Updated from 1999.io').push();

This works when testing with a single file. When I actually put it in a callback, I sometimes get errors when trying to push. The reason is when updating a post, the callback is called on 5 different files simultaneously. That's no problem when just writing files, but when they all try to push to GitHub over the network at the same time, some of them will fail.

I wasn't sure how to get the callbacks to keep track of what the others were doing, and for the time being I came up with a kludge. I only run the simpleGit commands for rss.xml (which gets written on every update), and using setTimeout I wait 5 seconds before doing so (since all the callbacks seems to complete in about 1 second, 5 seconds should be long enough for them to complete). My script is saved as a gist.

It's a interim solution, a kludge, it works well enough for my purposes, but I wouldn't recommend it for anybody else. However, GitHub does have an API where you can take actions without a local repository. Maybe that will work better? Stay tuned.

If you have no idea why you'd want a repository at GitHub, you can ignore this post blush

However, if you think you'd like to keep a copy of your site at GitHub, automatically pushing all updates, or even serving your site from there, you need to do a little bit of setup first.

  • On your server, clone your new repository. If your GitHub name is papascott and your new repository is named testing-rhino, the command is "git clone git@github.com:papascott/testing-rhino.git".
  • Next we need to create a SSH key so the server can push changes to GitHub without asking for a password. It's best to create a new key without a passphrase that is not used for anything else. 
ssh-keygen -t rsa -C "my GitHub key"
Enter file in which to save the key (/Users/you/.ssh/id_rsa): [Press enter]
Enter passphrase (empty for no passphrase): [Press enter]
Enter same passphrase again: [Press enter]
  • This will also create a public key as /Users/you/.ssh/id_rsa.pub. It will look something like this
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCt1Gad2wfIONTmXndX77
4Uwp8vU62Qr4p0VEzM29W6W7h4rx3vch7DsUk53jajxY4Eimrsybkxaka5v
KaasuV1cYN8nDbdRlaXjoefXNRi+JjEBv8i4zGhCmxKZpF1eWee2Up1Nyitsgl5
YOhEO2R0vO3uR1BMAgjefqazopXxVYFMut/ITTTvnK+alR4ngXQTAfTXcixZUg9ivXrMLwjSPQ+L3Py9/
Ql6RORdV0NH6uHqIWAdoHjfOerHek9+jo8DAXuSuf7IGK3qCp1pXXzpeahk0/QIwkNVph7tMI
LHa8rP7XKQpI2fo0JmEMi+SZ3sAc6+sz3o6Phz0yYCMro9 my GitHub key

(I've added line breaks for clarity, but it will actually be a single line.)
  • At GitHub we need to set this key as a deploy key for this repository, under the repository Settings/Deploy keys: Add deploy key, give a title, and paste the public key. Select "Allow write access", then "Add key".
  • Back on the server, we can test this by attempting to ssh to git@github.com:
ssh -T git@github.com
Hi papascott/testing-rhino! You've successfully authenticated, but GitHub does not provide shell access.

If you successfully authenticate, you should be good to go. In the next post I'll write the callback to actually copy the files every time we publish.

After a couple months hiatus, I'm getting back into 1999.io. Before I was using the domain 1999.papascott.de to stage articles for my main blog. But now I've decided to start over and keep 1999 as a separate blog. 

Resetting (clearing the home page and reverting the menus and templates) was pretty easy. Under publicFiles/users/papascott: 
  • delete chatLog.json
  • delete the folder misc and everything in it
  • restart the server

Now I've got a clean slate to post little tech notes like this one. blush The previous pages are still there just in case anyone is linking to them.