Network Coordinator--great gig.

It seems wrong that after such a long hiatus, I feel compelled to start this blog on a personal note, but so many of my friends around the country have asked:

Yes, as of June 30th, I’ll be retiring. If you read this blog, you probably also saw the announcement recruiting a new coordinator. For the past three years or so, I've been working weekends getting our 36' sailboat ready to do some long-term cruising. My contract as coordinator expires June 30th, and we plan to head south after that. The plan gets a little hazy (intentionally) after that. We will spend the winter in the Sea of Cortez and Mexican Pacific Coast. We'll get through the Panama Canal the following summer and head into the Caribbean after hurricane season.

So please, no symbolic “rocking chair” for me—it won’t fit on my boat! I can hear you saying “What, are you nuts!” Possibly—jury still out.

And how will the Network get by without me? Quite well, thank you. Ten years or so ago, we imagined a new paradigm for managing environmental data, and we wrote up a plan to build it out and drop it in place. To think that we’d just replace 15 years or more of legacy systems overnight is crazier than sailing off the edge of the earth. When I talk with folks outside our community (Homeland Security, USGS, Corps of Engineers, …) they are stunned by the degree of change we’ve already accomplished. As they say, turning a ship this size takes time. What I also know about ships is that the hard part is starting them turning. That part is done. Now it will be equally hard to stop it continuing to turn.

We’ve seen wholesale moves of business processes to the Network (EIS, WQX, …), and I don’t recall even one discussion of GHG data where the Network wasn’t a part of the picture. All over the country, people are using the same kinds of building blocks (Data standards, exchange templates, web services) to try to do what we’re doing, so we must have gotten something right. Every week, I get calls out of the blue from people who have stumbled across the Network on the web, and want to know how we did it.

It is true that progress isn’t always all we hoped. There are a lot of moving parts. Still, there are many processes where we’ve been completely successful and none where we’ve made no progress. Even better, the Network is doing things that we never imagined needed doing, thanks to some creative thinkers.

If you can manage a bit of travel, the coordinator job comes with:

• A dedicated community founded on mutual respect.

• Perhaps the most robust modern infrastructure voluntarily implemented by government on this scale.

• A ten-year and ongoing commitment from the environmental leaders of our country to make it work.

I highly recommend it. The next few years are going to be amazing.

ps: Don't forget to sign up for Chicago in April. I hope to see you there.

No Pressure, But...

The “Network Connection” has been quiet of late. I’m embarrassed to note that August is blank. If you’ve been wondering what I’d say next, so have I. Sometimes it takes an outside event to put thoughts in perspective.

Two weeks ago, I attended a conference in Baltimore, put on by the Air and Waste Management Association (A&WMA), on “Harmonizing Greenhouse Gas Reporting.” I heard a lot of presentations I didn’t much understand. Thankfully a lot of smart people are figuring out how to count greenhouse gasses—heck, I can’t even see ‘em! What was also clear was that there is a lot of “whistling past the graveyard” going on when it comes to data management. The audience in Baltimore was industry, lobbyists, and all levels of government (along with our friends at The Climate Registry (TCR) and other NGOs). They all knew that a large number of governments and NGOs will be collecting data and that:

• Each will define their own business processes, and have the authority to enforce them.
• They will respectfully disagree on many aspects of their programs, like certification and QA.
• The universe of reporters will partially but not completely overlap—a Venn diagram I can’t begin to draw.

And they are all hoping that someone will overlay on this a totally rational and simple “report once—use many” solution. Chet Wayland (EPA-OAQPS) and I talked about the work that we’ve been doing (and plan to do) with EPA and The Climate Registry. I didn’t hear anyone else even hint at how this might all work out. The few data issues raised were things like units of measure that we were working on 10 years ago. In fact the message was “we expect someone to make this simple”. By the end of the conference, it was more clear than ever that the “someone” in question is us-the Network.

Molly O’Neill said in 2008 (Nashville user meeting) that Greenhouse Gasses could be the “tipping point” issue for the Network. I can tell you now that no one else seems poised to step up and bring this all together. Today, I’m surer than ever that we will get up to the plate. If you haven’t heard, we’ve been working with EPA, TCR, select states and others to figure out how this might be managed. The Consolidated Emissions Reporting Schema (CERS) is one fruit of this effort. EPA has repeatedly reiterated its intent to collaborate with states and NGOs on data management, and to make the Network central to the solution.

So we’re going to get a chance! My first thought: Back in project management school, step one would be to get everyone involved to develop a unified business process. That is not going to happen. Prognosis: Failure. Solution: Run away!!

My second thought: We can do this. It looks a lot like projects we've already completed. With a bit of effort, I know we can put together a team and project that can get this done. The Exchange Network grew up on irreconcilable differences. The states and EPA didn't come together in the first place to do the easy stuff. Rule one is that everyone can be special. Network partners agree to respect differences. When participation is voluntary, the rules have to change. If anyone loses, they walk, and we all fail.

In my job, I spend a fair bit of time thinking about what makes the Network worth keeping (since that makes me potentially worth keeping). Technology? The Node is a pretty routine set of nearly obsolete technologies. It works for us, but it isn’t special in any way. It works precisely because we all agree to make it work. We have even developed a set of practices that help us exploit commonality and accomodate differences. If we can do anything that others can’t, it is to find solutions without losers.

For greenhouse gasses, we’ll have figure out how to respect at least 3 fundamentally different approaches to counting, and figure out where all the reporters overlap. If we do things right, we’ll let anyone who has to report emissions report them just once. We’ll make sure that everyone involved gets the data they need. And we’ll make sure that “nobody gets rolled” to make it happen. Probably impossible. Sounds like fun!

F5. So Easy a ...

In the leadership circles of the Exchange Network I’m seen as a technical type. When I hang out with the Network Technology Group (NTG) like I did last week, it is obvious that I’m Management. So when they said that I could build plug-ins for the OpenNode2, I was skeptical. The last time I actually wrote code that required compiling, I think it was in C. I never got the hang of Windows and C++. I have a pretty fair general knowledge of how this stuff works, but that’s it.

The NTG met in Kansas City last week. One item on their incredibly packed agenda was a demo of a tool to build plug-ins. In a matter of hours, I saw a plug-in built from scratch—literally dragging and dropping field from staging tables into the schema to implement the DMR flow. Not only that, but once the plug-in is built; the same tool can help you map data from your back-end system to re-use the plug-in. It really did look like I could do it!

You do have to have Visual Studio or the appropriate Java tool to make executables for your environment. After you’ve done all the dragging and dropping, you’ll have to put all the output files together and hit F5 to compile the whole thing. Funny—that is about all I remember about programming with Visual C. F5. So easy a manager could do it.

Note: This approach is part of a pending update release for OpenNode2. The node is free, and the plug-ins can be freely shared, but it does rely on a mapping tool that you’ll have to buy. It’s about 500 bucks. Then again, folks have been spending tens of thousands of dollars implementing flows. That’s a drop in the bucket if you save a couple of weeks work. If you can’t spring for the tool, you can still use the plug-ins that anyone else builds. You still have to have a source data system—no magic bullet for that, but it looks to me like this lowers the bar so far it is barely a tripping hazard!

Also from Kansas City: Look for an updated Design Rules and Conventions. The existing instructions on schema and flow design have been adjusted over the years to the point that they are no longer comprehensible. Read this—then modify it by what it says over there. If there are conflicts, do your best. We have learned a lot in the past 7 years. Now the NTG will be putting it all in one easy-to-read set of instructions. Watch for it. This will replace more than half of the instructions in our “library”. I’m not sure that means that a manager could design flows, but at least you or your vendor can do so with a little less frustration and confusion.

Forget Something?

RCRA Info V4 is open for testing. Congratulations are in order to the RCRA Info team. Astute observers of the Network will notice that it looks a lot like the earlier versions. No, you still can’t just dump your data on EPA’s doorstep and do updates by magic. Nor can you use your Node to get your data back out.

The new flow doesn’t solve some of the fundamental issues that have made this flow hard to do. So why did we build it like that? The old saw about swamps and alligators comes to mind, but we’re all about environment, and a swamp is a wetland. Wouldn’t be p.c. to drain it. So I’ll quote Dwane Young. Last winter when we met to talk about version 4, Dwane said “right now, my hair is on fire. I’m having a hard time thinking strategically.” What we saw in January was this: RCRA Info V4 was in production, and V3 was turned off. But there were no flow parts for version 4. No schema, no FCD, nothing. EN users were in a tough spot.

We agreed then that we had to put the fire out. The RCRA Handler & CME Version 4 flow opened for testing today. I can hear some people now saying “I know Mitch was there when we imagined what the Network would be. Did he forget? Why does he like this one?” There still isn’t any publishing. You still have to know a fair bit about what’s in the system to build new transactions.

So here’s why I like this. I used the term “we” a lot above. This feels like a “we”. We decided that this was the flow we could build in the time we had. We made a plan and teamed up to get it done. We can do better, and in time we will. Dwane and his whole team have made a commitment to adopt the exchange network. If you don’t believe that, take a look at the announcement that came out this morning to RCRA Info Network users. There’s more info in there about how to get help and what to do about problems than we have for any other flow. If you were waiting for outbound services and totally painless processing, just remember the last time you had too much to do, too little time. Be patient. Next time it might be your hair. You’ll expect the same.

If you see something new that doesn’t quite look like your vision of the Network, ask if it’s closer than the thing it replaces. I see a lot to like. OpenNode2, plug-ins that publish data automatically, Facility 3.0, EIS and WQX becoming “Network-only” systems… No one has forgotten what network we set out to build. It was a very long way from where we started. Some steps were big, some small, but we’ve taken a lot of steps in the right direction.

Dwane has promised to talk publishing as soon as the smoke dissipates. That doesn’t mean it will happen immediately. It means that we’ll work together to see what can be done.

What's Network?

I wrote last month that the Emission Inventory would be our very first nationally mandated report that was implemented exclusively on the Exchange Network. I’m not retracting the congratulations—and thanks to the emissions folks for working so well with us. But I wonder if we’ve been a little too strict with definitions. It could be that the Drinking Water folks got there way ahead of everyone.

As we started to build out the Exchange Network, we discovered that one size did not fit all (imagine that!). Someone realized that a desktop client can do all the reporting and data retrieval actions that some partners need. Nodes are important, and they can do lots of things that clients can’t (like publishing data, listening for incoming reports, and automating routine processes). Let’s be clear: Without a lot of nodes, there is no Exchange Network. However, when we start to count national reports, we decided long ago that using a client is right for some.

Enter the “web client”. CDX has created a client that is web-based. It uses Exchange Network methods and security, and implements all the network primitive methods. In it’s most “advanced” form, it has labels like solicit and submit and query, and you get to type in the URL of your target node, or look it up in ENDS. All very network-ish. Also totally inscrutable to an average user. So CDX is building customized clients that use the same back-end functionality, but hide "network-speak" from the user, and have controls like “QA EIS Data” in lieu of Submit, flowname=EIS, service=QA, …

Now to the point. The SDWIS State app builds a file that is clearly EN-compatible. Everyone uses a standard XML format. But most users are uploading reports via CDX Web. The user credentials, security, etc. all live in a different box. However, I’ve learned that behind the screen at CDX, CDX Web just hands the file off to the CDX Node processing mechanism—so it sure looks to the Office of Water like everything comes through the Network. Some folks have told us that it is still easier to check status and get error info through CDX web. It gets worse. The drinking water staffer can use CDX web without help. In many (most?) states that use nodes, he or she has to e-mail the file to the node administrator and ask them to submit it.

To distill the message: If you use our easy to use client and your credentials are stored in CDX Web, you are accomplishing the same thing, but no gold star. If you use NAAS credentials and a node or an inscrutable client that is less functional, then we’ll give you the gold star. This seems hard to explain to an end user. Maybe we shouldn’t try.

Here’s how we get to a more complete and robust network: We recognize that the benefit to EPA (for inbound reporting) is the ability to use a single channel. We go back to solid IT work--we build solutions for users that they can understand and use. It is our job to choose the right tool and reuse our investments. I think we agree that should include using the Network, but the users don't care, and shouldn't have to. If the client we gave the drinking water folks isn't network, replace it with one that is. It is OK if the user's can't tell the difference.

I’m not sure where this leads. Perhaps we check off the SDWIS federal reporting as “done”. If so, then here’s the next level for drinking water: Go beyond delivering mandated reports, and publish and share actual monitoring results. The health community will love that—they’ve been dying for seamless access to that information. By the way, the Association of State Drinking Water Authorities (ASDWA)is already working on something along those lines. More later.

ps—That will require nodes—maybe CDX, or maybe state nodes—probably both.

Give Chris the Hat!

Apparently the Wizard of the Wiki doesn’t blog! In my first-ever network blog, I challenged the EN community to find and make available the much-rumored Stylesheet that makes XML TRI files look like their paper counterpart. At the user meeting in Atlanta last week, Chris Staten (MS DEQ) got word of the challenge, and came through.

So it can be done. Now we just need to figure out how to share more easily. If it’s hard for me to find stuff—well, it is just too hard. If you have something that you think is worth sharing, I urge you to post it on the Wiki. Is that the ideal way to manage software distribution? No. Do we have anything else that easy for everyone to use today? No. I only wish we had so much stuff that managing it became an issue.

Speaking of Atlanta, It was great to see nearly all (both?) my regular readers there. If you weren’t there, where the heck were you? This may have been the best yet—the feedback received so far is great, but please respond to Kurt’s survey (anonymously). Tell us what was lame, or risk seeing it on the agenda again. For me, it was a case of too many good things. I missed a lot of things I wanted to see. If your presentation was one of them, I’m sorry, but it is bad form to wander off while moderating a session! There was nothing on the agenda that I didn’t want to see.

We introduced the meeting with a good news/bad news routine. The good news is that the network is behind the coolest application in decades. The bad news is that it only works in Georgia and North Dakota. We spent a fair bit of time talking in that session and throughout the week about the implementation matrix. I expect we’ll keep talking. This is a document that for the first time clearly lays out our progress in building a national network. The news isn’t all good, but there is plenty of room for hope. As soon as the matrix came out, people started explaining why things were better than it looks. A lot of the blank spots and inactive flows have already been fixed. If you saw what you think are errors, post comments on the Wiki. This tool could use some sharpening. Please remember that this is only counting national system flows. Not only are those one important line of network business, but they are the one we can measure.

Don’t get too hung up on the piece of paper, though. That matrix is so last year. Literally. Disregard the title at the top-the data was from CY 2008. Anything that didn’t happen in 2008 can’t ever happen in 2008. The Network governance has committed to fine-tuning this report and reproducing it regularly. I urge you to focus on the next edition—not the last. Use the Wiki to tell us what we might have gotten wrong, and to tell us what’s standing in your way. Implement or resurrect flows to change the colors in the boxes.

As usual, in Atlanta our tribal partners showed how they are redefining Exchange Network success to be what makes sense for them. I urge all partners to do the same. You get to define what success looks like for you, so we can help you get there. If you are "on the hook" for a flow that makes no sense to you, let us know that, too. The Network is about doing things smarter—not doing things that you don't need to do!

And to take it back to the top of the page, if you have a whole line of green boxes, look in them for any treasures you might be able to share. Your network is the network of the folks who are struggling. In the end, yours will share the same fate as theirs.

Way to go, Dennis

Way to go, Dennis!

If you have been around the Network much at all, you already know Dennis Burling. If not, well, Dennis is the IT manager in Nebraska. He’s been around the Network since it was not even a concept. Dennis has turned out the kind of “firsts” that don’t get noticed the way they should. He’s a “learn by doing” kind of guy.

Dennis and a few others (Dennis Murphy, Frank Catonese, Connie Dwyer, Chris Clark, …) got tired of talking about what the network might be, and went off to build it. Of course, they were way too early, so almost nothing of what they did survives. But everything we know about today’s Exchange Network those folks learned for us. Dennis also owned the first and only node ever to go to jail (his node was commandeered by hackers. If you want details, the cost of loosening his tongue is on you).

I guess you could say he is our leading authority on what not to do. So it is fitting that Nebraska was first out the gate with Facility 3.0. That flow was designed to incorporate everything we know about making a useful data flow. Full disclosure: I chaired that IPT with Pat Garvey when I was still with state government. Dennis took over for me when I moved on. Whether because of or in spite of my involvement, facility 3.0:

• Works exactly the same for everyone. EPA’s FRS implements the same services as any state or tribe.
  
• Provides ways to update a copy of the data without linking the systems. If Missouri wants Nebraska’s data, they can ask for it—to look at, or to store. Nebraska doesn’t have to even know which they did. It uses the same routines that update FRS, or allow any network partner to view the data.
  
• Includes full-featured publishing as an integral part of the flow.
  Allows you to “shop before you buy”. Very fast lightweight services allow you to preview what’s there—and then you can drill down for details. But only for the info you really want. That saves you plowing through data you didn’t want, and saves your partners the trouble of sending it.
  
• Incorporates the very latest in geo-referencing. Facility 3.0 includes GeoRSS. Today, you can use it to express a point location (Lat/Long) in a format that every GIS tool already understands. Later, you can plug in facility boundaries—without changing a thing.
  
• Answers that impossible question: What isn’t there? Facility 3.0 finally manages deletes, through a special service that allows copies of the data to be kept current.
  
  So, if you are designing any flow, take a look a Facility 3.0 (to be released to the Network web site soon). And join me in recognizing Dennis—for getting there first, and for learning for us all how to get there—the hard way.
  
  
  

Big Win for the Network

This week, I’m off to the annual Emissions Inventory Conference. For the air regulators and air scientists, this is mostly about air emissions, what they mean, and how to use them in making decisions. For me, it’s about two things: Greenhouse gasses and the new Emissions Inventory System (EIS). If you aren’t an air person, EIS is the new database that EPA is implementing to replace the National Emissions Inventory System (NEI).

If you have been watching Exchange Network progress, NEI was one of the earliest data flows. We have made steady progress over the years with states implementing NEI. However, the release of EIS is a really big deal for the Exchange Network. Here’s why:

Since the inception of the Network, we have assumed that one day, it would be “the way we do things”. But all the old EPA systems had a “way we do things”. Mostly, it was some variation on “log on, upload a flat file, download an error check report, and if you pass, promote the contents of your file to production”. The Network has always been the new thing, going head to head against something folks already know. ECOS just adopted a resolution saying that it was OK to talk about the Network being the only way—but EIS was already there!

Like NEI, EIS has no user interface. There will be three ways to get data into EIS:
• Use an Exchange Network Node to deliver it using the Network
• Use a desktop client to deliver it via the Exchange Network
• Use the new Network Web Client to deliver it via the Exchange Network

By he way, if you haven’t seen that last item yet, Kudos to Chris Clark at EPA—the web client is a nifty web site acts a lot like a node—but you need not install anything. You can use it to submit data, or ask others for data, using the Network. Just get a NAAS ID and you are on your way.

There has been an increasingly urgent conversation in the Network governance of late about how we get to our targets on national systems. Guess what? When the transition to EIS is complete, we’ll be at 100% for one flow. WQX was technically first to declare that the Network is the only way, but the Office of Water still does not require reporting into STORET as a condition for all monitoring. In contrast, if you implement the Clean Air Act (CAA) under a delegation agreement, Emissions Inventor data submission is a mandatory element of the program. .

I can hear the counterpoint: If someone uses a web site to upload a file, how is that a win for the Network? Looks a lot like the old way. As it says on the back of your lottery ticket, “There may be more than one winner”.
• EPA will use only he fully automated CDX processes to handle the file, regardless of the submission option chosen.
• EIS will use an XML format that complies with EN data standards.
• The EIS QA is Network friendly. It isn’t Schematron due to the complexity of some of the rules. However, it is possible to run QA on a file without submitting it—EPA need not waste their time trying to load bad data. Meanwhile, your node can test the file, get the report (almost immediately), and notify you when done. If your system is good enough, you can set your node to deliver on a schedule, and automatically upload if I passes QA.
• EIS will use the Consolidated Emissions Reporting Schema (CERS). CERS is an Office of Air, Climate Registry, and Exchange Network initiative to ensure commonality between collections of emissions and GHG data.
• EIS will grow the Network community. Remember CAA Delegation? Well, some regions implement CAA where states are not delegated, but in some states, CAA is delegated to local authorities. When all these folks use the EN to report, sharing data prior to the reporting cycle is just around he bend.

Sounds like a win to me. I’m calling it one, and many thanks for to the Air program for buying in. Let’s make it worth their trouble.

Data Exchange Opportunities—plain text format

Here are three “data exchange” opportunities that don’t require a schema, a Node, an FCD, or even any XML:

Share the load with Kurt and join me in the 21st century. If I can Blog, you can Wiki. We’d like to get updated and more complete information on network partners, and the Wiki is the easy way to do it—you get to write your own profile. The NPRG and support staff have already made templates on the Wiki based on what’s on the web site now. Just go to the Wiki and update your info. Then use it to find trading partners, or people you can borrow solutions from. When we get this up and running, maybe Kurt can take a break from calling you all and posting your info on the web. ps: Of course you can make stuff up for the Wiki—but you may be called on to do a presentation on it in Atlanta

Share your gripes with the NOB. Tomorrow, I’ll be on my way to a meeting of the Network Operations Board. Front and center on their agenda is how to clear the roadblocks that keep flow implementers from succeeding. We’ve often asked for your success stories (Keep ‘em coming, I love a good success story), but this time, we’d like to know what doesn’t work. Is there something about the Network or a particular flow that is awkward, hard to use, or just plain lame? Post a comment to this blog, by midnight Wednesday (April 1) and I’ll put it in front of the NOB. Then watch here for details on what will be done.

Last spring in Nashville, then EPA CIO Molly O’Neill talked about the buzz around Greenhouse Gas (GHG) data collection and management. She said it might be the “tipping point” issue for the Exchange Network. In case you don’t follow these things, EPA recently released it’s much-awaited GHG reporting rule. The rule contains a lot of different options (it’s over 500 pages, with another 500 of preamble to explain—in case you need a little bedtime reading). I won’t attempt to summarize even a little, except to say that it clearly recognizes that states, The Climate Registry, and EPA will all be collecting GHG data, and will need to share. It asks for comments within 60 days. There is no time like the present to check in with the people in your organization that are dealing with GHG data (believe me, you have them, even if you haven’t met). Find out what they think the rule means to them, and discuss their data exchange needs. If you or your future customers have expectations about how this should work, now is the time to get them on the table. In this case, use the Federal Docket process—I may respond to comments on the blog, but EPA will not!

Grants News—Don’t look here.

Last week, I wrote about some “non-grants”. I meant to comment on some great new work and continuing operations of the Exchange Network that is not receiving EPA funding. It is clear that many didn’t take it that way.

Let me reiterate: I have no knowledge of what applications were received or the decisions that were or will be made. I only know that folks are hearing about the results because a couple of applicants have chosen to share that information with me. All I can say about the actual ’09 grants is that decisions are being announced to the applicants, and you should hear the results soon—from EPA.

The Non-grants I wrote about are work that I know to have been done without a grant, some that probably isn’t eligible, or that I just made up. As far as I know, they were applications never written. If anything I described sounded like a grant you wrote or read, well—to paraphrase a novel, the account was purely fictional, and any resemblance to any grant, living or dead, is coincidental.

For the record, you will not find real grant news, or opinions about EPA’s grant decisions here-now or in the future. Making decisions on all the proposals received is a tough job—thankfully not mine. I will offer this opinion, though. While the grant program was and still is essential to building anything of this size, it is the work that is undertaken without grants that proves we are on the right track. In the end, it still has to look like a good decision—these folks are saying that it does.

It is good to know that people are reading—I’m sorry to find out through a misunderstanding.