| Sponsored by: International
Society of Regulatory |
Remarks on Peer Review: A Congressional Perspective Joseph C. Stanko, Jr., Esq. What I would like to do is talk about three things relatively quickly. One is how peer review fits into congressional action. Basically, I am not a scientist. I think we have a little different view of how peer review fits into the job of Congress and its obligations, although it is as important if not more important, than it is to you folks. I also want to talk a little bit about the process and some of the things I have seen in terms of what people say peer review is, and what it is not, then finally touch on some examples of how EPA has used, or not used, peer review, as well as the peer review, SAB-review process for the dioxin reassessment. First, let me tell you what I do and what my bosses do. I am counsel for the committee. I work for Tom Bliley, the chairman, and the members of the committee, advising them on environmental law matters. The committee basically looks at EPA with two different kinds of hats on. One is the oversight hat, which is where we examine EPA and other federal agency programs with regard to whether the agency is implementing the program the way that Congress intended, for the public policy result that was the impetus to pass the statute in the first place. The next one, which sometimes derives from the oversight efforts, is in a legislative context, where our oversight or other matters have led the members to conclude that we have got a gap here that should be addressed by a new statute. Now, I am a lawyer. I am not a scientist. That means, when it serves me, I will either say, oops, I don't know about that, I am not a scientist, or oh, yes, I know as much as those scientists, let's just keep plunging forward. Peer review is very important, because we need to know -- Congress needs to know -- what the risks are, whether the science that the agency uses -- whatever the agency is -- is the best science possible. Recently, I think one of the more important uses of peer review in risk assessment documents is what I would like to call the value points. Most of Congress, and basically all the country, realize that the Federal Government, despite the budget surplus that we have, does not have the resources to fix every single problem that comes in front of it. What Congress tries to do is what people do in their own lives -- that is to try to pick value points and design programs and review and make sure that the current programs are implemented to address those things that are major problems that need to be addressed for public health and welfare. Also, in distributing the limited resources of EPA or another agency, that those resources are being allocated in a manner that we are getting the best public policy result that we can. I think peer review of the agency's science and, many times, risk assessment really helps with that. By and large, as I said, I am not a scientist. I think the majority of the members are not scientists, although we do have a few that are physicians and other scientists, and we certainly have staff. By and large though, we are looking at the outside review and the agency's own technical review of where they are headed. I will now turn to process, which, from my standpoint, is an interesting question. What is peer review? I am a lawyer. I live in the world of defined terms. However, when people come up and testify before Congress, one of the interesting things that I have noted is the lack of a set definition, particularly sort of as you move a little bit farther outside the scientific world, of what is peer review. You have people saying, this is based on "peer-reviewed science," or we are "using peer-reviewed documents." What does it mean? Does it mean that somebody else who is a scientist kind of looked at it? Is it the type of peer review that you would expect before you get published in JAMA? There are just a lot of different definitions. In the debate on some of the other air standards that I have been involved with, this has been a real issue. In my mind, it sort of breaks down into about three general areas. The first is, who is doing the peer review? Is it scientists just in the client agency? I don't mean to demean that. There is excellent peer review that can happen internally. Certainly within EPA there is a lot of stuff that is trued up by that method. Is it other scientists that are in the Federal Government, in other agencies that happen to have an expertise, whether that be toxicology or cancer or the mobility of certain chemicals through a groundwater supply? The Federal Government itself has a lot of expertise that is available. Finally, obviously, the last potential parties, here, are outside, third-party independent folks who would look at and QAQC the science and the agency. If you are going to have true peer review and have the agencies get the benefit of that, I think to some extent, and certainly for the outside peer review, you should have everybody take a look. Scope is the next point. What are the peer reviewers being asked to look at? Has the scope been narrowed or is it too broad? Are they honed in on one particular topic that is going to miss another one that may be the most important one in determining whether to use sound science, or whether we are going to have a beneficial public policy outcome? That is another question that we have to ask when somebody comes up before us and talks about using peer-reviewed science. Finally, the issue of timing. You know, the best process in the world is not due process if you are jammed on timing or access to the scientific data in order to analyze it the way that a scientist must to provide meaningful comments to a reviewing board or the agency. So, at least from our standpoint, when somebody comes up and says, "we have gone through a peer-review process," those are some of the questions that we look at. With the exception of a very few statutes or regulatory procedures, peer review is not really defined. I think this has been a problem up on the Hill because someone will ask the question of a federal agency or an outside body saying, has this been peer reviewed? The answer the agency gives is yes, but they are talking past one another. We were fortunate on the Commerce Committee to have a couple of physicians -- in particular one from Oklahoma, Dr. Coburn -- who, during some of the Clean Air Act hearings, would go home with the actual JAMA articles and read them. The questions this guy asked were absolutely amazing. I think it really helped the committee understand, when EPA came forward and said, this science has been peer reviewed, what EPA meant by that and what that didn't mean. Finally, let me talk about a couple of EPA matters and I think that, again, the most important thing in designing a peer review process is taking into account all these elements and having the review occur at a time in the regulatory process that is appropriate, it allows for sufficient consideration and the agency can react to the information that it gets from the SAB or whatever reviewing body is conducting this, and change the policy if, in fact, that is what is warranted. There is currently an agency undertaking on the air act side called NATA, the National Air Toxics Assessment. For those of you who may not be familiar with this, basically it runs along the following steps. There is an air emissions inventory for the United States that EPA has. For this purpose they are using the one that has been trued up for 1996. EPA is running that through a model which has been used in the past to arrive at an estimate of ambient concentrations of air toxics for states, counties, the different political subdivisions. EPA will then take a model that is basically bits and pieces of other models the agency has used and come up with a risk indicator, which is an attempt to convey to the public what risk is being presented by this ambient concentration. This is a goal that the agency believes will be a cornerstone of risk communication in the out years. EPA is putting the model that it will use to derive the risk levels before the SAB for review. However, a bunch of EPA folks who, some time ago, sat in a conference room with me, told me that while they plan to do that, what they wanted to do was put up on the Agency Web Site not only the results of the inventories and the concentrations from the atmospheric modeling, but the results of the exposure model. They were going to rank states in terms of tons and risk and send this out to the world, and then send it to the SAB. I said, well, what happens if, despite -- I am not casting aspersions or alleging nefarious intent -- but if EPA issues the results and New Jersey comes up number one in air toxics? Commissioner Bob Shen from New Jersey is talking to The Bergen County Record, all the stories come out. You guys know the drill. Then, at the end of the SAB review, what if it turns out that, quite honestly, everybody realizes that something should be trued up or an equation should be different? You know, you rerun the model and New Jersey is number five or number 10. What are you going to do? EPA's response was, well, we can just go and change the Web Site. Now, I have spoken to state commissioners and state air directors. That is not a really satisfactory game plan. Oddly enough, the day after I had this conversation with EPA, there was a story about, I think it was the Department of Interior, ran a model about losing open space to development. Pennsylvania got an abhorrent score. There was a lot of press about this. A couple of months later, they realized there was a math error in the model. They reran it and there was essentially no net change in Pennsylvania. However, there were no news stories afterwards. It was really clear -- again, I don't want to attribute any nefarious intentions to the Agency, but it was just clear that the Agency folks hadn't thought about what it would be like to be a state commissioner and get these calls and have to deal with this for something that, in the end, may not be accurate. The ironic thing is that we weren't talking about a period of a year that this was going to be before the SAB. We were talking about four to five months for this peer-review process to come about. Fortunately, after some dialogue with us and with the states and with other folks, it appears that EPA is going to take the position that, well, the inventories are going to go up because everybody has sort of QAd and QCd that, but they are going to wait until SAB review to put up the risk analysis end of this. Again, I use this as an example that peer review is more than just getting in front of the right folks with the right amount of time. It is designing, in the regulatory process, that it takes place at the right point so that the Agency, in developing the final product, can take advantage of the scientific assumptions and the scientific critiques that come out of the peer review. It is slightly different because it really involves a legal standard, but the Agency's use of science in the chloroform rule making, which was returned recently, is another example of maybe not pursuing the best science possible or having the best science possible and not placing it in the appropriate context. In terms of the dioxin reassessment, you heard from other folks today arguments about the scientific nature of that. I would note, just from our experience from the procedural end, there may be some scope issues. In 1995, the reviewing body said, okay, EPA, go forth, change these sections. These other sections, they should be updated but maybe we don't have to see them. Well, when that came out in 1995, I am not sure that the SAB was contemplating that the Agency was to come back to them very close to the year 2001 for this. I would ask, given that, does the scope still hold or has both the document and the state of science become such that the SAB would be better served, and EPA and public policy, to have a broader scope for SAB review? Again, I am not a scientist, but it looks pretty much like that certainly wouldn't hurt things. Timing, again, is another issue. The Federal Register notice just came out this week. The final document has been available only recently which means that in order for commenters to get comments before the SAB, sending them in, having them distributed, having the SAB be able to review them, commenters only have two, maybe three, weeks to do that. That seems like a pretty short time period, considering that it has been a five-year run. I will just close with some notes regarding better serving public policy for peer review. You have got to look at who is doing it, what is the scope, what is the timing, and does it fit into the regulatory action in a way that allows the Agency to consider, in a valid manner, and respond to, the criticism. |