You are viewing jcreed

Notes from a Medium-Sized Island - [entries|archive|friends|userinfo]
Jason

[ website | My Website ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

[Jan. 9th, 2013|09:55 pm]
Previous Entry Add to Memories Share Next Entry
[Tags|, , ]

I got impatient with the arduino IDE because I can't even get it to display line numbers or jump to the line number of an error. Sheesh, what do I even want an IDE for if it can't do that. Pry emacs from my cold dead fingers etc. etc.

Anyway so I was trying to piece together instructions for how to run the good ol'-fashioned commandline compiler and uploader from the Makefile mentioned here, and after doing a lot of hide-and-seek with fiddly configuration parameters hiding in /usr/share/arduino/hardware/MaKeyMaKey/boards.txt and figuring out where to put them, I had an avrdude commandline that seemed like it should work:

$ /usr/share/arduino/hardware/tools/avrdude -C/usr/share/arduino/hardware/tools/avrdude.conf \
  -patmega32u4 -cavr109 -P/dev/ttyACM0 -b57600 -D -Uflash:w:/tmp/makey.hex


Only trouble is, it didn't. When talking over the usb serial port on /dev/ttyACM0, it found itself inadvertently communicating with whatever program was previously running on the board itself. So when the program running spat out a "4" every second or so as debugging information, the uploader would auto-detect some id field of the programmer as "4 4 4 4 4". Clearly wrong! Obviously it should somehow reset the board and prepare it for reprogramming, but... I thought it was avrdude's job to do that. I didn't understand what was failing.

I ran strace* on the arduino ide to find out what exactly it was execve'ing. Turns out, a commandline very much like the above. When I tried running the ide's command manually, it also failed. I was nearly running out of ideas for what the ide could possibly be doing that I wasn't (maybe weird ioctls on the serial port or something?) until I started nosing into the arduino ide source code, and found, in AvrdudeUploader.java, the following tell-tale comment: (the MaKeyMaKey board is indeed Leonardo-derived, I think)
    // need to do a little dance for Leonardo and derivatives:
    // open then close the port at the magic baudrate (usually 1200 bps) first
    // to signal to the sketch that it should reset into bootloader. after doing
    // this wait a moment for the bootloader to enumerate. On Windows, also must
    // deal with the fact that the COM port number changes from bootloader to
    // sketch.


Thus, I tried running
$ screen /dev/ttyACM0 1200
in a shell, and closing the connection immediately, and then running avrdude.

Which, of course, caused it to work.

I am a little shocked that this works as an out-of-band signalling mechanism. How does the board even know that a serial connection was opened and closed at a certain baudrate? I am betraying my ignorance of how serial protocols work at a low level, since there surely must be some negotiation happening that gives it away.

(*I ♥ strace soooo much, it can't be said enough.)
LinkReply

Comments:
[User Picture]From: aleffert
2013-01-10 02:59 am (UTC)

(Link)

The IDE is really terrible. Fortunately, it has a use external editor button in the preferences. Then you can just it as a giant run button.
[User Picture]From: jcreed
2013-01-10 03:00 am (UTC)

(Link)

but can I bind M-x compile to that run button? Huh? can I?
[User Picture]From: aleffert
2013-01-10 03:03 am (UTC)

(Link)

On a mac you could hook it up to the GUI scripting layer. No idea what your options are.
[User Picture]From: jcreed
2013-01-10 03:07 am (UTC)

(Link)

I think my favored option remains the "not GUI scripting layer" that I definitely have available to me, which doesn't require running a behemoth java program just to get a run button.
[User Picture]From: aleffert
2013-01-10 03:10 am (UTC)

(Link)

I mean, I think I agree in general, but something about the voodoo you know vs the voodoo you don't.
[User Picture]From: aleffert
2013-01-10 04:50 am (UTC)

(Link)

I think I also have a weakness for not breaking into abstractions until I've already absorbed more than I should of of minor aggravation.
[User Picture]From: jcreed
2013-01-10 03:01 am (UTC)

(Link)

p.s. emacs über alles!
[User Picture]From: aleffert
2013-01-10 03:04 am (UTC)

(Link)

p.s. emacs is weenies.
[User Picture]From: qedragon
2013-01-10 07:13 am (UTC)

(Link)

This is a little funny. The MaKeyMaKey shows up as a "USB serial" device, but that really just means that it speaks some particular protocol on the USB which, as it so happens, has good support for bulk-transfer of data in both directions and, more importantly, support in every operating system off the shelf in the past N years. (If you want to make your head hurt, go look at the USB packet formats and protocols; the particular stack of goo employed here is USB CDC ACM. See http://www.usb.org/developers/devclass_docs/usbcdc11.pdf , or better, don't.)

The "ACM" in all this is "Abstract Control Model" which basically manages all the metadata about the serial stream -- baud rate, control line state, etc. A typical USB-to-serial widget will take the ACM command which sets baud rate (as well as the width of a byte, the number of stop bits, and parity coding) and use this to adjust its on-board clock generator so that the right waveforms exit its pins. But really, it's just a particular packet on the USB (and happens to be tagged with a value whose mnemonic is SET_LINE_CODING). Similarly, there's a packet to toggle the "carrier" indicator -- called DTR on RS-232 devices (tag mnemonic SET_CONTROL_LINE_STATE; the Arduinos started(?) this trend by hooking the AVR's RESET pin to their USB-to-serial chip's DTR output pin, IIRC.)

If you want code, the particular hack you're running up against is this bit of the firmware: https://github.com/sparkfun/MaKeyMaKey/blob/master/firmware/Arduino/hardware/MaKeyMaKey/cores/arduino/CDC.cpp#L109 . If it looks like the host has done the requisite magic, write a magic value to the magic address 0x800 (hope nobody else was using that -- blech) and turn on the watchdog timer at 120ms, which means that in about 120ms the chip will reboot (unless somebody's calling wdt_reset(), in which case this hack won't work -- double blech). The bootloader checks for this magic cookie here: https://github.com/sparkfun/MaKeyMaKey/blob/master/firmware/Arduino/hardware/MaKeyMaKey/bootloaders/caterina/Caterina.c#L130 .
[User Picture]From: qedragon
2013-01-10 07:17 am (UTC)

(Link)

As a quick follow-on, you should be able to automate this with "stty /dev/ttyACM0 1200; echo -n '' > /dev/ttyACM0; avrdude ..." rather than needing to interact with screen.

Edited at 2013-01-10 07:18 am (UTC)
[User Picture]From: jcreed
2013-01-10 11:56 am (UTC)

(Link)

Naturally; I just already had screen in my bash history with a different baud rate at the time
[User Picture]From: jcreed
2013-01-10 11:55 am (UTC)

(Link)

I'm frankly alarmed how much this all makes perfect sense. I actually was at the point of concluding it had to be at the usb-to-serial layer that the magic was happening, but you've saved me a bunch of guesswork with your precise explanation, and saved me time hunting around in the firmware. Anyway thanks a ton. :) this low level hardware stuff is pretty fun after all, the occasional double-blech nonwithstanding
[User Picture]From: jcreed
2013-01-10 01:24 pm (UTC)

(Link)

While I've got your attention, though, I realize I'm still a little fuzzy about where boundaries are between "firmware" and "bootloader".

The CDC.cpp you mentioned was a file that I need to compile along with my program, which I had cargo-culted from
https://github.com/sparkfun/MaKeyMaKey/blob/master/firmware/Arduino/makey_makey/makey_makey.ino
which is definitely a "sketch", since its entry points are setup() and loop(), and relies on
https://github.com/sparkfun/MaKeyMaKey/blob/master/firmware/Arduino/hardware/MaKeyMaKey/cores/arduino/main.cpp
to actually provide a main().

So my Makefile had a line like
CXXSRC = $(TARGET).cpp $(ARDUINO)/HardwareSerial.cpp $(ARDUINO)/HID.cpp \
 $(ARDUINO)/USBCore.cpp $(ARDUINO)/new.cpp $(ARDUINO)/CDC.cpp \
 $(ARDUINO)/Print.cpp $(ARDUINO)/WString.cpp $(ARDUINO)/main.cpp

and all this stuff went into $(TARGET).hex and got uploaded with avrdude.

Is this "user program"? Is this "firmware"? Is it a mix of the two?

Am I right that https://github.com/sparkfun/MaKeyMaKey/blob/master/firmware/Arduino/hardware/MaKeyMaKey/bootloaders/caterina/Caterina.c therefore lives in "the bootloader" as distinct from either of these two, and resides somewhere non-volatile on the board that is not overwritten by avrdude during my routine reprogrammings? Is it possible to overwrite with avrdude in some other way? I appear to have a precompiled file
/usr/share/arduino/hardware/MaKeyMaKey/bootloaders/caterina/Caterina-makeymakey.hex
that I have already downloaded, but I also see the source in the same directory.
[User Picture]From: qedragon
2013-01-10 07:26 pm (UTC)

(Link)

I'd probably call it (and think most people would agree) "user program" and if pressed "statically-linked against support libraries". It is also "firmware", especially from the perspective of the program talking to it over USB, tho', which cares not a whit about the internals (it would be equally happy if you coded the whole thing in Verilog and had an ASIC fabricated).

The bootloader is a separate blob, and actually has some on-CPU facilities for protecting itself against errant programmers (For all the gory details, see chapter 28.1 among others of http://www.atmel.com/Images/doc7766.pdf ). It looks like their bootloader expects to always be entered at startup (suggesting that the chip has the BOOTRST fuse programmed, or that HWBE is programmed and they can pull on /HWB on startup.) and will always go to boot-loader mode after an external reset (so if you just whack the /RESET pin, assuming that's enabled!, everything should be golden; as you can see, there are a lot of configuration options over here. It's a bit of a mess.)

All of that aside, you'd be really hard pressed to get the chip into a truly un-programmable state, though you might need some external hardware to get it back to happiness if you've flashed a bad boot loader, like an SPI, JTAG, or, in the absolute worst case, parallel programming hardware. Sections 28.5 and on document more than you'd ever care to know about such beasts; I suggest not reading them and just enjoying knowing that options exist if you need them.
[User Picture]From: jcreed
2013-01-10 07:59 pm (UTC)

(Link)

wes you are setting a dangerous precedent by providing really excellent answers to my vague questions :P
[User Picture]From: jcreed
2013-01-10 01:51 pm (UTC)

(Link)

Actually wait wait wait whoaaaa. If the special magic thing that detects the SET_LINE_CODING USB packet is in the program that I upload every time to the board, does that mean I can brick it into an unprogrammable state if I mess around with the code doing that check? c.c