3D Hubs has recently made some controversial changes to their website which I would like to discuss. It’s currently in the experimental stage, but the impacts have reached far beyond a simple experiment. What I’m referring to is the new express checkout system which incorporates automatic hub selection.
I’d like to start by saying that I’ve always been a massive fan of 3D Hubs. I believe that the service is a great way of bringing together people with 3D printers and people needing parts printed. I’ve been a hub for almost a year now, and have earned a lot of money that I wouldn’t have without 3D Hubs. I actually think that the express checkout is a good idea, but it hasn’t been implemented very well so far.
Before I start talking about what I think of the whole concept, I thought it’d be a good idea to give a bit of context and make sure that everyone knows the facts about what 3D Hubs is introducing.
On August 31, 3D Hubs announced on their changelog that they would be experimenting with an automated checkout. Rather than the usual process where a customer selects a hub to print their order, an algorithm would select the best hub for the job based on factors like material offering, location, and hub ranking. 33% of US customers were subject to this experiment, and there was no option for the customers to opt out. Another interesting point is that only 50 pre-selected, strongly performing hubs in the US were eligible for the automatic selection. So essentially 33% of customers were taken away from all other hubs and divided amongst a group of 50 hubs.
This experiment also included the introduction of a new material selection system. Customers would select the material, colour, and layer height earlier in the process so that the algorithm could match them with a hub that used those materials. Below is an image of the checkout that was presented to the experiment subjects:
Some customers were presented with the new material selection system but not the automatic hub selection, and some were presented with both.
On September 22, another post was made to the changelog stating that 3D Hubs were launching the Automatic Hub Selection Experiment V2. The most important part of this experiment was the fact that the automatic hub selection became optional, so customers were able to proceed with a manual hub selection like before if they wanted to. This experiment is now running for all customers in the US, and uses the same 50 hubs that were selected for the initial version of the experiment. Here’s what the checkout looks like now. Note that it includes the new material selection system too.
So there’s the facts, now onto why people aren’t happy about these changes and what I think about it all.
I’m starting with this because it’s the easiest thing that could have been improved to make the situation better, and it would have circumvented all the shock from hubs wondering why they’re not getting as many orders all of a sudden.
Only the 50 hubs chosen for the automatic selection experiment were emailed about the changes. Other than the updates posted on the changelog, there was no communication from 3D Hubs to other hub owners in the US. I know that many people don’t read or even look at the changelog, and I think for such a big change that will most definitely affect business and take away potential customers, all hubs in the US should have been notified via email. It’s a small effort to avoid a lot of frustration.
I also think that after experiment 1, and even right now, it would be very beneficial for 3D Hubs to conduct online surveys with both customers and hubs about their opinions of the express checkout. They stated that they’ve done customer research and will be tracking statistics such as conversion to orders, but have they done research about how it impacts the hubs?
3D Hubs has acknowledged that they could have communicated better throughout the whole experiment and apologised for this.
Poorly conducted experiment
The idea of an experiment like this is to monitor the result of particular changes whilst avoiding any negative impacts that it may have until the results are certain. Statistics are collected and data is analysed to determine whether the change should be made universally throughout the service. 3D Hubs has done this successfully in the past, for example with the change from showing a hubs last active time to their average response time, and also with their introduction of printability feedback for customers. Both of these tests were conducted with only 50% of the target audience until the results were certain. Also, when instant order was introduced, it was done on an opt-in basis. People were happy with this because they were not forced to make any changes.
So starting with 33% of US customers for the experiment was quite reasonable. I strongly believe that doing tests like these are a good idea and a great way to improve the platform and add new features. However, extending it to 100% of US customers in my opinion was a mistake for these reasons:
- Errors in the algorithm
As the CEO of 3D Hubs stated: “One of the initial issues with Express was that the customer location wasn’t properly captured. This resulted in many orders going to just a few Hubs as the distance was no longer factored in”. Significant bugs like this should not exist in a system that has been implemented for 100% of customers in the US. I think that this should have been addressed before V2 of the experiment was launched.
- Unknown results
I don’t think that one month of testing with experiment 1 would have given enough conclusive data for it to be considered safe to expand to 100% of US customers. This was proven by the numerous bugs that were discovered as shown above. I think it would have been more beneficial to begin with a system that included all hubs, but went through testing for a longer period.
I’m also not sure how 3D Hubs plans to deal with new hubs that are just starting out with this automatic selection algorithm (that is if they end up being included). Surely they will have to be given a bit of an advantage in the beginning so that they can prove themself and accumulate some statistics from orders.
It just feels like it was far to early to roll out this experiment to the whole US user base.
- Low number of eligible hubs
Limiting all US orders that go through express checkout to a set of 50 hubs doesn’t seem fair at all. People have speculated that 3D Hubs owns or is working with these hubs and is financially benefiting from the increased traffic, but I don’t believe this to be true.
Gradually adding more and more to the list of eligible hubs would help to both spread the load of orders, and keep other hubs happy. I’m sure people wouldn’t actually have much of a problem with the express system if it included all hubs and distributed the orders fairly. If all hubs were eligible then the algorithm could select the best hub for the job. That way poorly performing hubs just receive fewer orders, and aren’t excluded completetly. I think this would be a much more fair way of distributing the orders.
It also seems surprising that 50 hubs are able to manage all the orders from those who choose express checkout (which is probably most) in the entire United States. That would be hundreds if not thousands of orders every day, so the selected hubs must be quite large.
- Taking away business from other hubs
This is the main point which is bothering everyone. The way that the user interface is designed prompts customers to proceed with the automatic selection. The button to proceed with a manual selection is quite out of place and is coloured to blend in with the background. Therefore the user interface is aiming to direct customers towards the 50 selected hubs, rather than looking at everyone else. This comes down to two things.
- The pre-selected group of 50 hubs should be larger. Hundreds, if not thousands larger. This will not only be more fair but will also allow 3D Hubs to collect more data.
- The user interface should not favour one option over the other. I think that both options should be displayed with equal significance, side by side at the beginning of the process. A customer could select whether they wish to complete their order with an automated selection process, or select a hub manually.
One issue arose where hubs with specific material names or colour names were not shown in the list of suitable hubs for customers because the names did not match up exactly with the customer selection. For example a hub offering “Ninjaflex” as a material would not be shown to a customer that selected a desire for TPU filament. Another more consequential example of this is if a hub offers “Prusa ABS”, it was not shown to a customer that selected ABS filament. 3D Hubs has addressed this by saying that they are building an entire new database of materials that will allow hubs to enter the exact details of their materials in a more specific way, down to even the brand. This will allow hubs to be matched to customers in a more organised way. You can preview the database here at the 3D Hubs material index. It looks pretty awesome to me!
The future of 3D Hubs
3D Hubs has made it very clear that the new express checkout system is being implemented for the benefit of the customers. But what they must remember is that the customers are only one half of their platform, and the hubs are the other half. So with new features like this, it’s important to consider how it will affect hubs too. In this case, I think they failed to consider how the hubs would react. The majority of hubs in the US are now receiving barely any orders as most of them are being distributed by the automatic selection system. I really hope that this is just a temporary issue until the number of eligible hubs grows rapidly.
It seems that 3D Hubs is headed towards a system more similar to Uber. A customer requests a ride, and Uber determines who would be best suited as the driver based on location and timing. Uber does the matching so that the customer doesn’t need to select a driver. As a side note, its interesting that Uber driver ratings don’t have an influence on whether a driver is chosen for a job. Instead, the drivers are required to maintain a particular average rating threshold which is determined by their number of ratings. With more ratings, the requirement threshold increases. If the driver falls below this threshold they risk being suspended. Maybe this would be an approach for 3D Hubs to consider.
I acknowledge the fact that most customers don’t want to have to deal with the 3D printing technology side of their order. They don’t want to worry about layer heights and selecting the hub that can best print their part. And for that, I think the express checkout will be great. The more customers that are brought to the platform, the better it is for all hubs. I like that the automatic hub selection removes the influence of factors like how well someone markets their hub and makes it a more even playing field in that sense. The number of orders you receive will be directly based off your raw metrics and how well you perform as a hub, not how pretty your display pictures and hub description are.
In terms of future improvements, I think the 3D Hubs developers could give some love to the hubs with more ways to track their own performance. I think that all statistics that are used to determine how a hub is ranked should be shown to the hub owner. It’s important to see how these statistics compare as well, so details showing what percentile you’re in for each statistic and a comparison with other local hubs would be really useful. This way each hub can see where their performance needs to improve the most, resulting in a better experience for the customers.
I hope that 3D Hubs remembers how important the small hubs are to its success, and isn’t gradually migrating towards more commercial hubs and print farms like some have suggested. Heading in that direction would be abandoning the core concept of community driven manufacturing and what I feel 3D Hubs was originally made to do – connect everyday people with a part to print, to everyday people who can print it.
Here’s some further reading for anyone interested in learning more about the changes discussed here: