Creating Map-Based Data Visualizations Is Getting Pretty Easy

By “pretty easy” I mean “even I can do it” - an exceedingly low bar. It appears that the secret is kepler.gl, New York City open data, and a little bit of python. Click here to play around with my map yourself.

I have been wanting to try and do a neat map-based thing for a while but did not really have a good project to use as a hook. That changed when I found out that New York City makes film permit data available as part of its (excellent) open data portal. My neighborhood is always having streets closed off for shooting and I was curious how it compared to the rest of the city.

Preparing the Data

Getting the data is pretty easy - just download it from the film permit data site. While that data set is good, it isn’t quite ready to be used. The data set has a lot of information about location - borough, zip code, streets closed, etc. What it doesn’t have is specific latitude and longitude. Those numbers are going to be key to placing the dots on the map so I needed a way to get them. Since there are tens of thousands of entries, I needed a robot to figure things out.

Fortunately there are robots that are good at this kind of thing. Reverse geocoding APIs allow you to submit address information and in return they will send you lat/long. They can even work with not-quite-address data, like cross streets and zip codes.

In order to make things work, I wrote a python script to open the file from the City, send the useful bits to the geocoding API, and add the lat/long to a new file. I decided to use the Mapbox geocoding API because Mapbox is great.

It turned out that the data needed some cleaning before the API would give me what I needed. The first issue was that I had forgotten how to deal with the fact that the first row was a header row. Since I am lazy and there was only one input file I just deleted the row in the input file and manually pasted it back into the output file at the end.

in terms of more substantive issues, sometimes there were many locations associated with a single permit, so I decided to break the entries up by location. This code breaks up multiple-address entries into individual entries:

addresses = list(row[6].split(',  '))

The for loop then works through each address to make it its own entry in the output file:

for address in addresses:

If there were multiple addresses there were also multiple zipcodes. I admit that I cheated here. Instead of figuring out how to map the multiple zipcodes to the multiple addresses, I just picked the first zip code and used it for all of the addresses. My hope was that it would be close enough:

zip_split = list(row[13].split(', '))
first_zip = zip_split[0]

After playing with some test data I realized that at least 30% of the addresses I sent were not recognized by the geocoder. Helpfully the geocoder returned some address in West Africa when it was confused, so it was easy to find the errors. After playing around a bit I figured out that stripping the ‘and’ and ‘between’ from the addresses made the geocoder more accurate:

address_no_between = address.replace('between', ' ')
address_fully_stripped = address_no_between.replace('and', ' ')

I also realized that the order of terms mattered for the geocoder. Specifically, putting the street names at the end of the term seemed to help:

geocoder_entry = borough + ' New York City ' + first_zip + ' ' + address_fully_stripped
print("GC entry = " + geocoder_entry)


#looks up the address and pulls the lat and long
response = geocoder.forward(geocoder_entry)
counter += 1
collection = response.json()
lat = collection['features'][0]['center'][1]
#writes it to the csv
#row.append(lat)
long = collection['features'][0]['center'][0]

That greatly reduced the number of errors from the geocoder, although it did not eliminate them. In order to try and salvage something, if the geocoder returned a bad lat/long I resubmitted the query with just the first street name:

short_geocoder_entry = borough + ' New York City ' + first_zip + ' ' + short_address
print("Short GC entry = " + short_geocoder_entry)
response = geocoder.forward(short_geocoder_entry)
counter += 1
collection = response.json()
lat = collection['features'][0]['center'][1]
long = collection['features'][0]['center'][0]
row.append(lat)
row.append(long)
output_writer.writerow(row)

That got the accuracy rate high enough for me to stop worrying about it.

Using the Data

Once I had a correctly formatted data set it was time to move over to kepler.gl. At this point things became comically easy. The steps were just:

  1. Go to kepler.gl
  2. Upload the output file (make sure you have re-added the header row)
  3. Add a filter tied to the field “StartDateTime”

That is the whole process. Kepler automatically recognized that I wanted to show the locations over time and added the timebar to the bottom of the screen (click here to play with the real one):

Dotmap

If I wanted to add hotspots over time (instead of individual dots) that was as simple as adding a new layer:

Heatmap

That’s basically it. I am excited to use this visualization for more things and I hope this post helps prove to you that pretty much anyone can make real looking map visualizations.

Earlier this week OSHWA announced that it would not be holding the 2019 Open Hardware Summit in Shenzhen, China. The stated reason for canceling the Summit is that China’s new foreign NGO law created logistical and bureaucratic barriers to holding the Summit in China. This reason is absolutely true – OSHWA was not going to be able to overcome the barriers created by the law.

However, there is also a second reason that I am wary of holding an event in China: China is currently holding one million ethnic Uyghurs in concentration camps, primarily because they are ethnically and religiously different from the Han Chinese majority. While OSHWA did not discuss the Uyghur situation in the Summit announcement, I believe that it is important to flesh out this concern for the community. I also believe that the open community should also consider this situation when it thinks about bringing major events to China.

The Situation

Uyghurs are a predominantly Muslim ethnic minority who primarily live in western China. China has currently placed perhaps one million Uyghurs in what are commonly described as detention camps, re-education camps, or gulags. Uyghurs are in these camps because of their racial and religious identity. For Uyghurs not placed in camps, the Chinese government has ordered another one million Chinese citizens, known as “big brothers and sisters,” to live in the homes of Uyghurs and report their activity back to the government. Uyghurs are also being forced to shave their beards, eat pork, drink alcohol, and generally partake in activities that violate their religious beliefs for the purpose of distancing them from those beliefs.

Speaking out about these camps - even for people outside of China - can be dangerous. China has routinely placed the relatives of Uyghurs outside of China in camps as punishment for speaking out. This makes drawing attention to and discussing this practice highly fraught in general, and and all but precludes responsible discussion of them within China.

Even if one were inclined to draw a moral equivalence between China’s treatment of Uyghurs and any number of activities undertaken by the U.S. government today, the combination of massive human rights abuses and punishment for activists who dare to raise concerns about the abuses has no equivalent in the United States or any other western country that I am aware of. That makes the situation worthy of discussion within the open community.

OSHWA and Uyghurs

The Uyghur situation presented a potential problem for holding the Summit in China this year: should OSHWA hold its major event in a country engaged in this kind of activity? While an easy question to ask, I will not pretend that this is necessarily a question with a straightforward answer.

OSHWA is an organization organized around a community of open source hardware. Human rights issues are not obviously part of that primary mission. At the same time, the open community strives to be inclusive. OSHWA is rightly proud of the Summit’s code of conduct and the Ada fellowship. While these things are not directly related to making open source hardware in a narrow sense, we recognize as a community that it is important to fostering an inclusive and vibrant open source hardware ecosystem.

In light of that, how should OSHWA evaluate human rights abuses in potential Summit host countries? Is it OSHWA’s place to speak out on human rights issues? If so, how should it do so and what kind of legitimacy does it speak with? To the extent that OSHWA has a community mandate to speak about open hardware issues, does that mandate extend to human rights issues?

Furthermore, OSHWA has many community members in China. What does it mean for their safety if OSHWA begins speaking out against human rights abuses there? If OSHWA comes out against what is going on with Uyghurs, should agreeing with OSHWA’s stance on that issue be a prerequisite for supporting open source hardware? What would that mean for OSHWA supporters in China who support the government’s activities? If OSHWA was to get involved with this situation, how would it handle other non-open hardware-related situations in the world? If OSHWA is worried about the situation, is it better to go to China and engage with people about it or simply walk away?

These are mostly answerable questions, but they are questions where reasonable people could disagree. In light of that complexity, OSHWA decided to take a path of constitutional avoidance with regard to the Summit. Because OSHWA could decide not to hold the Summit on relatively non-controversial procedural grounds – the foreign NGO law effectively made it impossible to hold the Summit in China this year – OSHWA decided not to wade into the Uyghur situation as well.

What are the Ramifications?

I think this was the responsible decision by OSHWA. That does not mean that the community - as opposed to OSHWA as an organization - should necessarily ignore the Uyghur situation. In fact, I believe that the open source hardware community has a responsibility to understand what is happening in a country that is so important to it.

I certainly would not have been comfortable participating in a Summit held in China with one million people in concentration camps. That discomfort would have been compounded by the fact that even pausing for a moment to recognize the existence of the camps at a China-based Summit would have exposed the organizers - Chinese and non-Chinese alike - to potential punishment from the Chinese government. That would make it irresponsible to do so.

I also am not suggesting that a relationship with China is binary. I continue to use products made in China and engage with Chinese companies. It should go without saying that raising this issue is not an attack on the Chinese people, and that I enjoy and hope to continue to grow my relationship with many members of the Chinese open source hardware community. In fact, I hope that there are open hardware events in China as part of Open Hardware Month in October.

Nonetheless, holding the Summit would have been a special type of engagement. OSHWA chooses the city for the Summit in order to highlight the importance of that city for the community. Special types of engagement should bring with them special types of responsibilities and reflections.

OSHWA does not have to directly grapple with how the Uyghur situation interacts with the Summit this year. However, there are many other organizations in the open source hardware community (and the larger maker community) that are considering holding events in China this year. Assuming they can find a way to navigate the NGO law (or that the NGO law does not apply to them), I hope that they will at least pause to consider what their event might mean in a country with a million people in concentration camps.

This post originally appeared on the OSHWA blog.

This blog post is an update for the OSHWA community about the 2019 Open Hardware Summit in Shenzhen, October as Open Hardware Month, and how OSHWA will think about Summits going forward.

The tl;dr version of this post is:

  1. OSHWA will not be holding the Open Hardware Summit in 2019
  2. OSHWA will be encouraging locally-organized events and gatherings across the globe as part of Open Hardware Month this October (email us at info@oshwa.org if would like to host one!)
  3. OSHWA will shift the Summit to the spring starting in 2020. The Summit will also be held in the same city for at least 3 years starting in 2020.

There is a lot to unpack here, so let’s get to it.

2019 Open Hardware Summit in Shenzhen

At the end of the 2018 Summit OSHWA announced that it would be holding the 2019 Open Hardware Summit in Shenzhen, China. Shenzhen has a vibrant local community of open source hardware enthusiasts. Many members of the OSHW community were also very excited for the opportunity to travel to a location that is so central to manufacturing innovation.

Unfortunately, in 2017 China implemented a law governing the activities of non-Chinese NGOs operating in China. This law created a number of bureaucratic hurdles for organizations like OSHWA that were interested in holding events in China.

Among other things, the law requires OSHWA to find a local Chinese Partner Unit (CPU) willing to act as our sponsor for the Summit. CPUs can only be certain types of organizations, such as universities or registered Chinese NGOs. Companies cannot serve as CPUs. The CPU must also be willing to undertake a significant number of bureaucratic steps to officially register the event and coordinate with local authorities. In addition to the process required of the CPU, OSHWA itself would have to undertake a significant and burdensome number of steps to collect, verify, and provide paperwork to Chinese authorities (see this article “Reams of Paperwork: Preparing Documents to Get Official Status in China” for a sense of what is involved).

OSHWA has spent the last few months trying to identify a suitable CPU. We have been unsuccessful, and do not have confidence that we will be successful in the future. Furthermore, even if we were able to find a suitable CPU, OSHWA cannot justify the time and resources required to comply with the various filing requirements associated with the law.

As a result, OSHWA decided that it was better to cancel the 2019 Summit now, before speakers and attendees had made commitments and travel arrangements.

That being said, OSHWA is still committed to supporting the OSHW community. That is why we are pairing this announcement with two additional announcements.

October as Open Hardware Month

OSHWA has traditionally supported October as Open Hardware Month. Open Hardware Month is an opportunity for the community to hold local events, hackathons, and documentation days as part of an international movement.

OSHWA wants to take this opportunity to expand Open Hardware Month events. We will work to provide resources for the community to create to local events, aggregate information to make it easy to find events in your area (or know that you need to organize one), and collect stories, video, and images of the events as they occur. These events will not be OSHWA run or carry the formal OSHWA name. We believe that Open Hardware Month will provide us an opportunity to shine a light on open source hardware events happening around the world. It will also provide an opportunity for local communities to raise their hand and be recognized by the global OSHWA community. Please email info@oshwa.org to learn more and volunteer to be involved.

Spring Summit 2020

Cancelling the Shenzhen Summit and focusing on Open Hardware Month will also allow us to shift the Summit to the spring. Over the years a number of Summit participants have told us that the spring is generally less crowded with events and obligations, so this shift should make it easier for more community members to attend.

Starting with the 2020 Summit OSHWA also intends to commit to a single US host city for at least three years.

For the past year the OSHWA board has been debating two alternative paths for the Summit. The first path would continue the pattern of moving the Summit every year. The benefits of this path is that it allows the Summit to come to the many different communities that support OSHW. The costs of this path are that it makes the Summit more expensive to operate because OSHWA needs to spend time and resources learning a new city every year. Switching cities also makes it hard to capitalize on the enthusiasm of local attendees in order to convert them into full community members.

Conversely, the alternative path is to commit to a single city for multiple years of the Summit. The benefits of this path is that it allows OSHWA to run the Summit significantly more efficiently and makes it easier for community members to plan. Holding the Summit in a single city allows OSHWA to grow the number of attendees by turning opportunistic local attendees into more permanent members of the community. The cost of this path is that it prevents us from moving the Summit to all of the communities that support OSHW.

After significant discussion, OSHWA has decided to adopt the single city approach. This decision was easier because we paired it with the expanded Open Hardware Month. We believe that Open Hardware Month will help fill at least part of the gap created by a stationary Summit.


While none of these decisions are being made lightly, OSHWA believes that combined they allow us to create a rhythm that is more supportive of the vibrant OSHW community. Open Hardware Month in the fall will shine a spotlight on all of the local OSHW communities around the world. The Summit in the spring will provide those communities a single place to come together and meet in person.

As always, OSHWA exists because of its community and we want to hear from you. Please let us know what you think in the comments below or in the forums.

This post originally appeared on the Create Refresh medium page.

The internet does not consist entirely of a handful of large, static platforms giving creators access to mature technologies such as audio and video. If it did, Article 13 of the Directive on Copyright could be an interesting approach to challenges posed by the online distribution of material protected by copyright.

Instead, the vitality of the internet comes from a dynamic array of websites that help connect creators and audiences with an ever-evolving array of expression and technology. Article 13 fails to imagine this internet as it exists today, not to mention the internet that we want to evolve two, five, or ten years in the future. As a result, it may push the internet in a smaller, more centralized, less creative direction.

Article 13’s filtering provisions are written with an eye towards technology that large platforms have implemented to analyze traditional media such as sound and video. Even setting aside the imperfect nature of this filtering technology — and it is far from perfect — the provisions overlook the fact that the technology did not simply appear when these platforms were created. Instead, these filters are the result of decades of work and hundreds of millions of euros of investment.

Requiring that type of investment as table stakes prevents new platforms from entering existing areas. Perhaps more problematically, it also erects significant barriers to new platforms that bring new technologies to creators.

Until recently I was the General Counsel at the 3D printing company Shapeways. Shapeways was founded in the Netherlands and gives hundreds of thousands of creators access to cutting edge 3D printing technology. This is not the desktop technology you may be most familiar with. Instead, creators use Shapeways to access printers that can cost hundreds to thousands of euros to print steel parts, gold jewelry, nylon dresses, and production-ready medical braces. Shapeways then empowers creators to market their works to a worldwide audience of fans and customers.

Unlike audio or video files, there simply is not a technology that can accurately identify 3D files in order to identify their content. Furthermore, there is no central database of all existing physical objects to compare files uploaded to Shapeways against. This makes it nearly impossible for a platform like Shapeways to comply with a requirement to filter or identify uploads for possible infringement.

Shapeways has been a leader in developing innovative licensing agreements to connect large rightsholders with Shapeways creators. It also has a robust process to allow rightsholders to request models be removed from the site and expeditiously complies with such requests. This system works well in the vast majority of cases. Nonetheless, Shapeways creators are sometimes targeted by unscrupulous claims where the purported rightsholder has a tenuous — at best — legal basis to request a model be removed.

Today’s legal structure allows some creators targeted by such unscrupulous claims to fight to keep their creations available online. However, if faced with increased liability for the activity of users, Shapeways and platforms like it will often err on the side of removing content in the face of any colorable claim. This is because providing any individual user the opportunity to challenge accusations could risk the viability of the entire platform.

Unable to comply with statutory filtering requirements and mindful of increased liability, under Article 13 platforms like Shapeways would be forced to institute a review process that would fundamentally alter the nature of the service. Instead of being available at internet scale, 3D printing would once again be beyond the reach of most creators. Only creators with orders large enough to justify an expensive manual review of each file would be able to place an order. Platforms looking to bring similarly cutting edge technologies to creators across Europe would be forced to make similar calculations.

Article 13 creates obligations that new platforms simply cannot meet. It fails to understand today’s platforms beyond a handful of large players, and fails to anticipate a richer, more diverse future where creators have access to even more tools. The Directive on Copyright should focus on empowering creators, not limiting the tools available to them.

Michael Weinberg is the new Executive Director of The Engelberg Center on Innovation Law and Policy at NYU Law. He was previously and for many years the General Counsel at the 3D printing company Shapeways, founded in the Netherlands and which gives hundreds of thousands of creators access to cutting edge 3D printing technology. The views expressed here are his own.

OSHWA Supports Design Patent Clarity in Amicus Brief

This post originally appeared on the OSHWA blog.

OSHWA has just filed an amicus brief in a case regarding design patents. OSHWA urged the court to uphold a rule that a design patent covers only what the patent itself says it covers. This rule allows everyone to understand what is and is not protected by a design patent. A clear understanding of the scope of design patent protection is particularly important for open source hardware creators who share their design files for use and modification by others because they need to know when a patent would – and would not – apply to their design.

The Case

The case in U.S. court, called Curver Luxembourg SARL v. Home Expressions, Inc., is actually about furniture patterns. Curver applied for a design patent on a wicker pattern similar to one found in ancient Islamic designs. The pattern looks like this:

Image Pattern

Design patents don’t protect abstract designs as represented in all things at all times. (Copyrights do that.) When you apply for a design patent you need to identify the “article of manufacture,” the actual thing that embodies the design. Curver initially failed to identify the thing that embodies the design, but eventually identified a “pattern for a chair.”

Curver’s designation of a “pattern for a chair” is important for what comes next. Another houseware manufacturer, Home Expressions, started selling a basket with a wicker pattern similar to Curver’s. Curver accused Home Expressions of infringing on Curver’s design patent. Curver believes that now that its patent for a “pattern for a chair” has been issued, the patent should be interpreted much more broadly to cover baskets, or any other object embodying the wicker design.

OSHWA’s Amicus Brief

From OSHWA’s standpoint, it does not really matter if the patterns on Curver’s and Home Expressions’ baskets are the same or not. What is important is that Curver’s patent is for the design embodied in chairs and baskets are not chairs (this is a hill I am potentially willing to die on). Curver should not be able to select arbitrarily or strategically the thing that embodies its design in order to get the patent, and then turn around and apply the patent well beyond the scope of that selection in the real world. Our brief asks the court to adopt a rule preventing that kind of behavior.

Regardless of what you think about the design patent system more broadly (and there are many opinions about it in the open source hardware community), the system can work only if patents give notice of what they cover. The “article of manufacture” (in this case, a chair) is essential to providing that notice because it shows how an otherwise abstract design applies to a particular object. It also places a reasonable limit on the scope of a design patent’s protection. Home Expressions should have been able to confidently ignore a chair-based patent in designing their basket.

The trial court agreed, and found for Home Expressions, but Curver has appealed the case. OSHWA has filed an amicus brief urging the appellate court to uphold the trial court. Our brief is in support of the rule that patents should be read to cover what they say they cover – and only what they say they cover.

This is important to the open source hardware community in at least two ways. First, creators cannot avoid infringing on existing patents if they do not have a way to understand what those patents do and do not cover. The patent system works only if people can figure out from patents themselves what those patents cover. This is important for maintaining a healthy environment for open source hardware creators to share design files with others without exposing themselves or other creators to unknown risks.

Second, some open source hardware creators rely on licenses to impose openness obligations on future users of their hardware. Those creators cannot understand when the openness obligations apply to future users or how far those obligations extend if they cannot understand when the design patents included in the license are being used.

Curver actually took the fairly unusual step of opposing our request to file our amicus brief. Fortunately, the court recognized that the open source hardware community could be impacted by the decision in this case and denied Curver’s attempt to keep us out.

OSHWA will continue to keep an eye on this case and provide updates as they develop. We would also like to say a big thank you to Kyle McLorg, George Laiolo, Erik Stallman, and Jennifer Urban at the Samuelson Law, Technology, & Public Policy Clinic at Berkeley Law for representing OSHWA in this case. They drafted the original brief, as well as the argument that convinced the court to accept it over Curver’s objections.