July 3rd 2016 Flutter Update
almost 8 years ago – Sun, Jul 03, 2016 at 09:21:49 PM

Hello friends,

I know you've been wondering what Flutter has been up to, so here's the news.

• Pro boards have arrived. They work great, aside from the aforementioned frequency offset issue.
• I've discovered the cause of the frequency offset issue. Our manufacturer incorrectly substituted the radio crystal for several different parts and this is causing the strange behavior.
• Seeed is willing to replace any offending crystals, but this would add more than a month to our delivery time.
• We're working with Seeed's local office to analyze the boards, replace some of the crystals, and determine a course of action.
• We will either implement a one time factory calibration or will rework all of the boards with US sourced crystals. There will be no more customer calibration step required.
• Once this rework is completed all shipments will begin. They will include the Pro boards. Top backers still get their shipments first, and those boxes are all ready to go as soon as we've got approved boards.

Check out the awesome Pro boards. We've got 1600 now and I'm loving them. That said, I've got a big update below. Please check it out:

June/July Update

I made a mistake.

I thought we were ready for shipping. I was excited to see the hardware in your hands, to silence the nay-sayers, and to move past the struggle that has been manufacturing 4000+ boards.

Turns out, production isn't over yet.

I made another mistake by delaying this update. I know you all want to know what is happening on a timely basis and this update got pushed back too far. I was waiting for a better plan and I know that's not a good justification.

That said, I'd like to let you know what's been going on.

I discovered three weeks ago why we've had this frequency offset on our boards. Despite informing me that they had made no substitutes, our manufacturer, Seeed Studio, used several substitutes for the 40MHz crystal that the radio uses as a frequency reference. Some of the substitutes were a similar part from another manufacturer, and some seem to just be counterfeits. They are now remedying the situation, but it does mean shipping is being pushed back.

I learned about the substitutes by taking photos of the crystals under my microscope and emailing them to Epson, the manufacturer of the crystal I specified. This is what they said:

This is not an Epson part. Part marking has a backwards R. Manufacturer is River Eletec Corp. See below for link to their website. http://www.river-ele.co.jp/index_en.html

This is an Epson part.

Not an Epson part, counterfeit. Has an extra 0 in the frequency which is not our part marking structure. Compare to the part marking of Crystal 2 & 4.

This is an Epson part.

Not an Epson part, counterfeit. Has an extra 0 in the frequency which is not our part marking structure. Compare to the part marking of Crystal 2 & 5.

It took me some time to figure out how to react to this news. I've got $60k in hardware funded by personal debt and if the hardware is bad it would be a catastrophic setback to this project. I have no experience with this kind of a situation and as a single individual working on this project, I had no idea if my manufacturer would be willing to help. I wasn't even sure if the crystals were responsible for my frequency woes. It was difficult not to be paralyzed by fear.

I could have ignored the problem. We have a software fix that lets you calibrate boards against each other. And while I would consider that an acceptable choice if we had no other options, the more I thought about it the more I disliked the out of box experience this created for the customer. You should not have to calibrate your boards. This was never part of the plan, and the calibration step makes certain things, like mesh, more difficult. Armed with the knowledge that the crystals were bad, I decided to see if replacing them would solve the offset problem.

I spent a weekend improving my frequency error detection software. The originally hastily written code took a while to get a reading and produced inconsistent results. After improvements, I had a fast, repeatable test that would calculate the frequency shift for any board, as compared to an RF reference.

Then I spent a weekend measuring error, replacing crystals, and re-measuring error. My sample size was small, but it seemed pretty clear to me that replacing the crystals on poorly performing boards brought them in line with my expectations.

Finally I felt like I was armed with enough information to reach out to the manufacturer. While I am very unhappy with the use of unauthorized substitutes, I was pleased to see that they are willing to replace any boards that don't meet my spec.

Conveniently, they have a US office about an hour from my house. Inconveniently, they recently decided to find a new location and the one remaining employee has lost his Visa. He leaves the country July 7th.

The last week has been a mad dash as I've handed off a little over 1000 boards for analysis, visited their office to deliver a test laptop, and visited one more time to deliver a test desktop when the test laptop seemed to be failing.

Here's what we've learned.

Out of 549 Basic boards:

• 169 have counterfeit "Epson" crystals
• 354 appear to have Epson crystals
• 26 have crystals made by River Eletec Corporation

Out of 576 Pro boards, 576 have crystals made by River Eletec Corporation.

I reached out to River Eletec for more information on these crystals, but they aren't sure which batch they are from or what their specs are exactly.

While I've been inclined to throw out all the crystals and ask our manufacturer to replace them with the correct part, I wanted to see if the crystals we have might actually be appropriate substitute. We'd be more flexible in production if we had multiple suppliers we could use, and we could ship your your hardware a little sooner.

This is what the data looks like as compared to a reference board using a US sourced Epson crystal.

 

You can see that in the worst case, two boards from this batch could be offset by more than 30KHz from each other. Since we have a 23KHz wide signal, the boards would miss each other entirely without some kind of calibration. You can also see that for any one crystal type, the distribution is better. If all the boards used River Eletec crystals, or US sourced Epson crystals, or even the counterfeit crystals, we would be fine. The radio chip compensates for small errors automatically; the manufacturer of the radio chip recommends a crystal with +/- 10ppm of error, or +/- 9150Hz at 915MHz.

I now have to decide if we will implement a one time factory calibration that measures the board's error against a known reference and saves it in EEPROM, if we will send all the boards back to China to have their crystals replaced with US sourced Epson parts, or if we will do both.

I'm leaning towards at least implementing the calibrate to EEPROM option, so we can get all boards to agree very accurately on what frequency they are transmitting on. If we do that, we might be able to accept some or all of the boards as is.

Next weekend I'm going to work on saving the offset data to the EEPROM, and I'll make a final determination on what to do with the boards we have. I'm not interested in rushing this. As much as I really want to be DONE with the manufacturing stage and move into documentation and software improvements, it will be much worse for this project if we ship hardware that has problems.

I'm learning a lot from this process. As I continue to move this project forward, I'm learning how I can do this better in the future. I appreciate you all giving me the opportunity to learn this, and I hope that what I deliver to the community is worth the frustration many of you have experienced as a result of these delays.

I love you all,

Taylor

May Flutter Update - Shipping begins!
almost 8 years ago – Sun, May 22, 2016 at 03:46:45 PM

Hello Flutter Backers,

In our last update, I shared that I was tracking down an issue with the new boards, and was waiting to ship until I had a handle on the issue. Well, I've been working on the software fix, and will be pushing that to our git repo in the next few weeks. Meanwhile, it's time to start shipping.

I'm going to ship partials starting with the highest reward tiers. That is "It's not -not- Skynet", and "A Challenger Appears". Cool note about the latter group - they will have the electronics necessary to build Scout.

Packaging supplies arrive this week, so the next step for me is to double check the necessary shipping addresses and start loading up boxes. If you are a backer at the above levels, look for an e-mail from me this week to confirm your address. Shipments for the top 53 backers will all go out before the end of June. These are partials that won't include the Pro boards, but they will have Flutter Basics, Explorer boards (formerly R/C Shield), Starter Kits, Breakouts, and for "A Challenger Appears," Remote Control boards.

While there is a lot more engineering work to be done both on the software and hardware sides, my primary focus for the next month will be on shipping these 53 boxes and on prepping for the larger shipping batch.

If I don't run in to any hiccups, I will be shipping the first batch of boxes on the weekend of June 14th, and will post a quick update when that happens.

That's all I have for the Flutter update, but I'd like to promote a friend's Kickstarter. If you love LEDs and want some easy WiFi-controllable LED strips, check out Flickerstrip. This is Julian's second successful Kickstarter after the ReFlowster oven controller, and the campaign has 12 days left. I've seen the hardware in person and it looks great so pledge now if you want awesome LED goodness.

Love,

Taylor

Testing update
about 8 years ago – Mon, Apr 25, 2016 at 10:39:39 PM

Greetings humans,

If you cannot read, check out this sweet section of squiggles below:

• I can now clearly characterize the unexpected behavior I have observed with the new boards.
• I have collected data from 11 new boards so far, and have plots of some of the data below.
• I believe that software improvements can eliminate the issue. The behavior I am observing may be normal for hardware like this.
• I have more tests planned for this weekend and expect to determine the solution soon.
• Once the solution has been tested, I will begin shipping.
• Explorer boards should arrive to me this week.
• Pro boards should ship out to me within a week or so.
• After shipping is under way, I can begin to focus on software improvements and documentation.

Complete update

I wanted to share an update on the board testing progress. If you recall from last update, I have discovered some unexpected behavior with the boards from the latest production run, and I don't want to ship anything out until I understand what is going on.

In the past few weeks, I've worked to improve my data plotting capability so I can collect accurate data about the frequency offsets of the boards I have. At this point I don't have a definitive root cause to the problem, but I now have a reliable way to measure the issue and characterize the behavior. I will be working to nail it down more this weekend, but I wanted to share a few plots of the data I have so far.

You can see from the first graph that the maximum error I have found is 23.5kHz, and that an error on the order of 15kHz is common. I will definitely be collecting samples from more boards next weekend, and also need to see if the error changes with channel. I did do a freezer test to see if the offset is affected by temperature, and so far it appears that it isn't.

The boards may just need to have a simple calibration value programmed in to them. That would be easy and would quickly fix it if the offset is fixed and static. I do have some ideas about how to handle the condition where the offset drifts. Regardless of the specific solution, it seems clear to me that we can solve the problem with software. Once I have a well tested solution, I will begin shipping units.

I've also been told the Pro units will be headed my way within the next week, and Explorer boards will be delivered to me this week. That means that pretty much all of the rewards can ship soon. :)

 Just for fun I took a plot with a wider frequency range too.

Thanks for reading.

Love. <3

April Flutter Update
about 8 years ago – Mon, Apr 11, 2016 at 10:59:08 PM

Hello!

Have a bulleted list of things:

• Starter Kits have arrived
• Pro Boards are in progress
• 350 Flutter Explorer boards shipped to me last Friday
• I am investigating an issue with frequency accuracy on the Flutter Basic boards
• I will begin shipping Basic boards as soon as I get to the bottom of the frequency issue
• I still love you <3

Pro Kits are still being manufactured, but we got some pictures recently.

My car is full of Starter Kits because I am running out of space in my apartment for boxes.

See the following terrible image for some idea of what the kits contain.

Importantly, I discovered an issue with some of the Flutter Basic boards where the radio frequency is off by +/- 50 KHz or so. This results in inconsistent radio transmissions between boards, as our signal is only ~23KHz wide. The CC1200 radio chip does a good job of compensating for some errors as it can align itself with any signal it detects, but if the error is too much the signal gets missed.

There's a few possible causes, and I am working hard to figure out the best solution.

Possibility #1: The manufacturer swapped out the crystal I specified for something else that isn't good enough. That would be bad, and I would need them to re-work all the existing boards if that happened. I have contacted the crystal manufacturer (Epson) to see if this crystal's markings (E15CA) match the crystal I specified (boards I made myself have several different markings). I have also ordered some of the correct part from Digikey and I will see if swapping only the crystal has a significant impact on performance.

Possibility #2: The correct crystal was used, and this kind of frequency offset is to be expected. I will have to fix it in software with some kind of calibration routine, but I don't think that will be a problem.

I wrote some software for Flutter that shifts the frequency a few kHz at a time and attempts to read packets from a transmitter with no offset. It saves some data into arrays and then spits out pretty graphs via the USB port so I can figure out WTF is going on. Screen caps look like this:

 As you can see, these two boards have pretty different offsets (it may be worse than actual here, as I was not using a known good board on the transmitter end). I am still refining my test methodology, but the basic plan is this:

• Ensure testing software is functional and repeatable.
• Collect offset values from a 10-20 random boards from this batch.
• Take 5 of them and swap out the crystals for known good crystals from Digikey.
• Test again and compare new frequency offset numbers to original results.
• Then place all the boards in the freezer till they are chilly and test again.
• Compare the effect of temperature and the effect of replacing the crystals.
• Try to make sense of it all.

As I said, I have also reached out to the crystal manufacturer to determine if the marking I see on these chips match expected markings, as the datasheets don't say. It does kind of seem like the markings are Epson markings, so we may need a software calibration routine for all these boards. That's in many ways preferable as it would mean I could ship a lot sooner.

I will be working hard to get answers here, as I desperately want to SHIP these things but I will not ship product without a planned solution. If the fix is just software, I will ship them ASAP with a note of the impending software update.

I've been doing some other things too. The manufacturer wanted to test antennas for the Pro boards for functionality, so I wrote a cute program that lets them tell a Pro board what signal with an antenna looks like and what signal without an antenna looks like, by looking at signal strength. It also let me take the following picture, where the LED changes color based on distance from a transmitter:

That's all the technical stuff, but if you've made it this far you obviously care about this project so I'd like to share a little with you. This project is extremely important to me. It has been difficult for me to manage all the work required to get this far and there is a lot more work to be done to get functioning product in everyone's hands with good software, documentation, and examples. I must deliver though. I have a vision for my own life that requires that I figure out how to run this business come hell or high water. Flutter is an opportunity for me to take my ideas and get them in people's hands while giving me the creative freedom I dream of. I have other ideas and I want to make them real. To do that, I must deliver this project. And we're close.

Thanks for hanging in there.

Love,

Taylor

Flutter Basic boards have arrived!
about 8 years ago – Mon, Mar 21, 2016 at 12:37:39 AM

Wooooo!!!!!!

Ze boardz, zey have arrived. :)

Okay, I don't know about you but I'm really excited. I've got twelve freakin boxes of Flutter Basics taking up space in my tiny apartment and it's a pretty cool feeling.

These are JUST the Flutter Basic boards (and their associated USB cable). Most rewards also require the Flutter Pro and the Explorer Board (previously "RC Shield"), so until those arrive I'm only going to be fulfilling the small number of rewards I can ship out now (Hello World backers who did not get in on the May shipment).

As I mentioned in a previous update, our Chinese manufacturer no longer offers shipping services, so I have to ship all this stuff out myself. That is a BIG JOB and shipping from the US is more expensive than shipping from China, so I won't send out duplicate shipments. That means that until the Flutter Explorer boards and Flutter Pro's arrive, few shipments will go out. (EDIT: Backers of the highest level "It's not -not- Skynet" will receive partial shipments ASAP. I gotta show them some love.)

I am still communicating with the supplier about testing of the Explorer Boards so those have not shipped to me yet. I also have been told that samples of the Pros are being built now, but I don't currently have more information there.

For now that's all the news that I have, but I hope you're as excited as I am to see this moving along! I'll keep y'all updated as new information comes in.

As always, please share your questions, comments, and accusations in the comments. ;-)

Love,

Taylor