1. Is it new? (is this just re-hashing something we already knew, in a slightly different way, or is it a new discovery or is it using a new method to more definitively tell us something we previously only suspected)
2. Is it true? (is the methodology sound, were the results significant, was it a large and representative sample)
3. Does it matter? (will this impact how we treat a disease for many people, or how health care is delivered, or does it advance our basic understanding of an important disease)

How to read and understand a scientific paper: a guide for non-scientists at ViolentMetaphors.com - Thoughts from the intersection of science, pseudoscience, and conflict, is a good article on the subject:

Here's a summary:
Reading a scientific paper is a completely different process than reading an article about science in a blog or newspaper. Not only do you read the sections in a different order than they're presented, but you also have to take notes, read it multiple times, and probably go look up other papers for some of the details.
As you read, write down every single word that you don't understand. You're going to have to look them all up (yes, every one.

1. Begin by reading the introduction, not the abstract.

2. Identify the BIG QUESTION.
Not "What is this paper about", but "What problem is this entire field trying to solve?"

3. Summarize the background in five sentences or less.
What work has been done before in this field to answer the BIG QUESTION? What are the limitations of that work?

4. Identify the SPECIFIC QUESTION(S)
What exactly are the authors trying to answer with their research? There may be multiple questions, or just one. Write them down. If it's the kind of research that tests one or more null hypotheses, identify it/them.

5. Identify the approach
What are the authors going to do to answer the SPECIFIC QUESTION(S)?

6. Now read the methods section. Draw a diagram for each experiment, showing exactly what the authors did.

7. Read the results section. Write one or more paragraphs to summarize the results for each experiment, each figure, and each table. Don't yet try to decide what the results mean, just write down what they are.
THINGS TO PAY ATTENTION TO IN THE RESULTS SECTION:-Any time the words "significant" or "non-significant" are used. These have precise statistical meanings. Read more about this here.
-If there are graphs, do they have error bars on them? For certain types of studies, a lack of confidence intervals is a major red flag.
-The sample size. Has the study been conducted on 10, or 10,000 people? (For some research purposes, a sample size of 10 is sufficient, but for most studies larger is better).

8. Do the results answer the SPECIFIC QUESTION(S)? What do you think they mean?

9. Read the conclusion/discussion/Interpretation section.
What do the authors think the results mean? Do you agree with them? Can you come up with any alternative way of interpreting them? Do the authors identify any weaknesses in their own study? Do you see any that the authors missed? (Don't assume they're infallible!) What do they propose to do as a next step? Do you agree with that?

10. Now, go back to the beginning and read the abstract.
Does it match what the authors said in the paper? Does it fit with your interpretation of the paper?

11. FINAL STEP: (Don't neglect doing this) What do other researchers say about this paper?

Source: How to read and understand a scientific paper: a guide for non-scientists at ViolentMetaphors.com