I think the simplest way to look at it is to say that an aversive is something an animal will work to avoid.  This ties in with Paul Chance's definition of negative reinforcement, but I do want to point out that he says it is "ordinarily something the animal tries to escape or avoid."   The reason I have spent time on definitions here is because I wanted to see if there was complete agreement that negative reinforcement always involves an aversive and that an aversive was always a bad thing.  I think I was also hoping to find that negative reinforcement did not require an aversive because when I use negative reinforcement well, it doesn't feel like I am using an aversive.  But instead I found that I needed to broaden my own understanding of what defines an aversive and recognize that it is just something an animal will avoid.  Different things will be aversive in different situations and the definition of "an unpleasant stimuli" covers a broad range from mild annoyance to pain.   I think this is important for clicker trainers because we want to use negative reinforcement with an aversive that is on the milder end of this spectrum.  

     Another reason I looked up these definitions is because I have found that of all the quadrants, negative reinforcement seems to be the most confusing to precisely define, both in a technical sense and in the application.  The name seems to be part of the problem, more so than some of the other quadrant names.  Perhaps it is because people do use the term negative reinforcement without understanding it and it seems to get mixed up with negative feedback.  I do sometimes hear people using the term negative reinforcement and I have never heard the average person talk about using positive punishment or negative punishment in any context except operant conditioning.

    Some people have such an automatic response to the word "negative" that they can't get past that part. And some people think that if it is reinforcement, that means it is pleasurable to the animal (as opposed to reinforcing a specific behavior).  Some trainers find it difficult to think of training in terms of removing something.  It does not help that in application, negative reinforcement is often confused with positive punishment.   But I also think one reason it is hard to understand is because there is such a range of ways to apply negative reinforcement and the look and feel of negative reinforcement can be very different depending upon the stimulus.

    I want to remind you of the variety of ways in which negative reinforcement can be applied or occurs in everyday life.   In the previous article, I listed a number of ways that negative reinforcement happens and I am going to include them again here.  These examples come from a few different web sites (http://www.princeton.edu/~yael/LearningCourse/Notes/Examples.doc, http://www.utexas.edu/courses/svinicki/ald320/negrnf.html) and from my notes from various speakers and books. 

    Even though there is a huge amount of variation in these examples, the list above shows that all applications of negative reinforcement have one thing in common:  a behavior increases if it precedes the removal of a stimulus. The stimulus could be as mild as having the sun in your eyes, to having a headache, or taking on an unpleasant task.  In negative reinforcement, the stimulus that is applied is called the negative reinforcer.  For most of this article,  I am going to use the word "stimulus" to refer to the negative reinforcer or aversive because it seems the most neutral term to use.  I do at times use the other terms when it seems more appropriate so I just wanted to clarify that the negative reinforcer and stimulus are the same thing. I do sometimes use the word "aversive" if that seems more appropriate or to make a point.

    In the previous article I wrote about the connection between negative reinforcement and positive punishment. If you did not read that part, I suggest you go back and do so as it is very relevant to using negative reinforcement as a clicker trainer.   As a brief reminder here, I will just say that if I use negative reinforcement in a training situation, I often have to both apply and remove the stimulus.  The negative reinforcer that I choose could also be acting as a positive punisher if the application of the stimulus decreases the behavior that was happening when I apply the stimulus.   In simpler terms, when I use negative reinforcement, I have to be aware that I could be punishing one behavior at the same time I am reinforcing another. 

    The fact that the same stimulus can act as both a negative reinforcer and a positive punisher becomes important when it comes to choosing the stimulus and evaluating my training. It will come up again when I write about poisoned cues and escalating pressure. I wanted to mention it now because it relates to another aspect of negative reinforcement which I did not discuss in the previous article and this is the escape-avoidance aspect of negative reinforcement. 

    In some applications of negative reinforcement, the subject can learn to minimize the exposure to the stimulus (escape) or escape from it entirely (avoidance) and this is why negative behavior is sometimes called "escape-avoidance learning."    Looking at the examples above, you can see how the negative reinforcer could be escaped from or minimized (by responding sooner) or avoided entirely by responding before the stimulus is applied.   I was doing some reading in Paul Chance's book,  "Learning and Behavior," and there seems to be some controversy over how negative reinforcement works because if negative reinforcement is effective, the subject can avoid the stimulus entirely.

    Avoidance can happen if the subject finds a way to predict the application of the negative reinforcer before it happens. In this case, it would seem as if negative reinforcement was not being used at all, but avoiding the negative reinforcer is still driving the behavior.     This can make it hard to see when negative reinforcement is still influencing behavior because the original stimulus is not presented at all in order to maintain the behavior.  I think this has some implications for horse training and evaluating our use of negative reinforcement so I want to explain a bit more about "escape" and "avoidance."

The Escape-Avoidance Aspect of Negative Reinforcement

    The easiest way to understand this is to pick an common example.  Let's say that my mother wants me to wash the dirty dishes so when I come into the kitchen, she starts nagging me to do them.  If I do them when she nags me, then this is a successful example of negative reinforcement.  Doing the dishes (the desired behavior) removes the nagging (stimulus) and I am now trained to do the dishes when she nags me.  So far, so good, but depending upon how I feel about her nagging, this could change in several ways. 

    If I continue to find her nagging more unpleasant than doing dishes, then I will continue to do them when she nags. Over time I might start doing them after less nagging because I have learned that the sooner I start, the less nagging I have to hear (this would be escape).  And if I still find any nagging unpleasant, then I might start to look for other indicators that I need to do the dishes and do them before she starts nagging.  I might learn to do the dishes as soon as I see them in the sink and therefore escape from any nagging at all (this would be avoidance).   But this is assuming I could not avoid the situation entirely. In some cases, avoidance leads to the subject avoiding the stimulus condition under which the aversive occurs.

Is it negative reinforcement, positive punishment or both?   

    When I am using negative reinforcement, I am using a negative reinforcer as a stimulus to prompt a change in behavior.  This negative reinforcer could also be acting as a positive punisher.   Being able to identify when I am punishing one behavior as I increase another behavior is an important step in learning to use negative reinforcement well.  In the previous article, I gave some examples of using negative reinforcement with and without positive punishment and some guidelines for evaluating your own training. 

    In that article I spent time on the difference between negative reinforcement and positive punishment because I wanted people to see how they could move away from training that was based on stopping "bad" behaviors to training that was based on training "good" behaviors. For most people, that is an important step, but it is also important to learn how to use negative reinforcement as a teaching tool where it is being used to train new behaviors by prompting, guiding or giving feedback to the horse.

    I am going to share some of the things that have worked for me and other people I know.  Clicker training is a very flexible tool and there are going to be other ways to do things. I am hoping that sharing what I do will give people ideas to get started and they can move on from there.

    Guidelines to becoming a more positive reinforcement based trainer

Combining Positive and Negative Reinforcement

    As I noted in the previous article, most traditional horse training uses negative reinforcement, but negative reinforcement has some drawbacks.  When clicker training was first being applied to horses, a lot of the training was done using only positive reinforcement and the model of dolphin training was the one most people used. This was true of dog training too. A lot of early dog training emphasized free shaping and the animal was at liberty and able to choose to participate in the training or not.  This worked very well for training many types of behaviors but was not quite the same as training a horse to be ridden.  Unless horse people wanted to throw out everything and start over, this meant horse people had the interesting challenge of figuring out how to combine a positive reinforcement based training system with a negative reinforcement based training system. 

    The easiest way to do this is to use traditional training methods, often based on pressure and release, and add positive reinforcement on top of that.  So when the horse gives the correct answer, the trainer responds as usual (by removing the "aversive" stimulus) and in addition, she clicks and reinforces.   When training a new behavior the trainer might click and reinforce every correct effort so the horse is getting reinforced through negative reinforcement and positive reinforcement. One the behavior is more established, the trainer might click and reinforce only the best efforts and just use the reinforcement provided by the removal of the stimulus to reinforce other efforts that were correct but not of the same quality. 

    This is what Alexandra Kurland refers to as "piggy-backing" and a lot of her early work is based on combining John Lyons' training methods with clicker training.  For a lot of people, adding a click and treat to what they are already doing is a good way to start using clicker training.  This allows new clicker trainers to start with a system that they already know and as they get more familiar with clicker training, they usually start to see places in their training where they can make changes in how they use negative reinforcement as well as where they can take advantage of the power of clicker training.

    Alexandra Kurland's own approach to clicker training has evolved over the years. She has put a lot of time and effort into coming up with better and better ways to educate people about how to use negative reinforcement in such a manner that it does not require aversives, or shut down the horse's behavior.  She has also come up with new ways to train behaviors using only positive reinforcement alone that combine well with more traditional approaches.  This has led to a form of horse training that has the benefits of both types of operant conditioning.  If you are interested in learning more about her methods, I have written a lot about them on this site and she has her own web site (www.theclickercenter.com) as well as numerous books and videos. 

    But some people come to clicker training with a horse that has had bad experiences with traditionally applied negative reinforcement or that doesn't respond to it well in the first place.  Others come to clicker training with a strong commitment to only using positive reinforcement. Or they might be learning how to clicker train horses after having previous experience with clicker training other animals.   For these people, starting with a more positive reinforcement only focused approach works better.  There can be a lot of emotional baggage associated with past training or using negative reinforcement and by staying with positive reinforcement, the horse and trainer can build a solid relationship and learn more about each other while still training new behaviors.

    These days I find myself in the interesting position of being in the middle of the these groups.  On one hand, I can understand that there are some people for whom piggy-backing the clicker on to their own style of training is the easiest way to start clicker training and they are happy with the results. On the other hand, I can see the value in trying to shift to a more and more positive approach or even starting by avoiding any use of aversives in training.  I think this is an area we need to explore more and I am thrilled when someone is willing to put the time and energy into training behaviors with the emphasis on positive reinforcement alone.  At the same time, I am a practical person and I value training that is effective and not stressful for the horse, regardless of what quadrant I am using.

    This means if you put me in a room of clicker trainers who say you can only use positive reinforcement, I will find myself arguing the value of negative reinforcement.  But if you put me in the middle of a group of negative reinforcement/punishment based traditional trainers, I will be the one arguing for using positive reinforcement.   This is not a question of there being a right answer because every horse and person combination is different and we all have different goals.   That is one reason I think it is important to become educated about our training options and find a system that works for you. Good trainers are always changing as they learn new things and we are all at different stages in our journey. 

    I started out as a traditional horse person and many of the horse skills I have learned over the years involved negative reinforcement, and even punishment.  When I found clicker training, I was thrilled to learn there might be ways to train horses with a strong focus on positive reinforcement.  I started out by doing some shaping work that was only positive reinforcement, but I also added the click and treat on to what I was already doing which was basic dressage training with some trail riding and jumping.  I had mixed results until I started working with Alexandra Kurland and learned more about how to use negative reinforcement as a clicker trainer.   She uses negative reinforcement as part of her training plan but the emphasis is on the finding and rewarding the good moments.  She also does not escalate when using negative reinforcement but gives the horse time to find the answer. If the horse gets stuck, she breaks it down into simpler pieces.   I learned more about using Alexandra's techniques for combining negative and positive reinforcement and I am very pleased with our progress.  

    In the last few years, I have started playing around with variations on her exercises and trying some of my own things and this has helped me understand how to become an even better user of negative reinforcement as well as how flexible clicker training is and how the same behavior can be trained in so many different ways, with varying combinations of positive and negative reinforcement.

   The rest of this article is going to focus on the progression from a negative reinforcement based training system to a more positive reinforcement based training system.  I am writing this with the traditional horse person in mind who is now trying to apply clicker training to their horses and needs a bit of a "road map" for how to get to the point where they are using the full potential of clicker training, or to find the place where they are comfortable with the blend of positive and negative reinforcement that they are using.   I think this is a process. I think there are some people who can jump in with both feet and only use positive reinforcement and maybe they will find some ideas in here too.  I am hoping this will give people a way to evaluate where they are and give them some idea of the next level that is around the corner. Sometimes people don't know what is possible because they don't know how to look for it.

Guidelines to becoming a more positive reinforcement based trainer: Evaluating your use of Negative Reinforcement

    I am going to start at the beginning with the simplest way to add clicker training to your program.  This does not mean that these are the first steps you would take to clicker train your horse. I always recommend that people start with targeting and train some simple behaviors before they try and teach or improve current training issues.  There are a couple of reasons for this. First, it takes a new clicker trainer a while to develop good timing and a sense of how to shape behaviors so that the horse doesn't stall out and get frustrated or get stuck at an intermediate step along the way to the finished behavior. It is better to practice and learn these skills on behaviors where there is no previous baggage or emotional investment.   But even as a more experienced trainer, I start with simple things because I want the horse to be motivated to play the game before I ask for any behaviors that might be difficult either physically or because of past emotional baggage.  If you are not sure where to start, Alexandra Kurland has foundation exercises that teach practical skills to both horses and handlers.

    Once you are past the beginning stages and your horse understands about the click and treat, what do you do next?  You can take the foundation exercises and use them as building blocks for a lot of other behaviors (this is the subject of another article, coming later).  Or you can look at what you already do and see if it can be improved by adding positive reinforcement.   This usually means you are adding positive reinforcement on to a negative reinforcement based program and I think that is fine, provided you follow a few basic guidelines.   The nice thing about adding positive reinforcement is that it will make you better at using negative reinforcement and over time, if you are conscientious and observant, your program will shift toward more emphasis on positive reinforcement. 

    You might be wondering if there is value in becoming better at negative reinforcement if you are going to shift to a positive reinforcement based training system, but I think there is. I think that if we are going to work with horses in any manner where we are connected to them, either by a lead rope or sitting on them, then we owe it to our horses and ourselves to become very sophisticated users of negative reinforcement.  While I might be able to set up my training and teach new behaviors using only positive reinforcement, when I ride there are going to be times when I end up using negative reinforcement.  And there might be situations where using negative reinforcement is the most effective way to train.  My goal is to have a horse that understands about negative reinforcement so that I can use the minimum amount of stimulus to get a response in most situations.  But I also want my horse to be ok with those times when I do have to use a stronger stimulus so that in case it encounters extra pressure, it does not panic.

     I mentioned earlier that just adding positive reinforcement to a training program based on negative reinforcement can make a significant change by helping trainers become more skilled at using negative reinforcement as well as by making the training experience more pleasant for the horse.  There are some other advantages to piggy-backing clicker training on to an existing training program as compared to using a traditional negative reinforcement based system alone.  Some of these are:

1. Horses learn faster because the click clearly marks the right moment and the reinforcement makes them motivated to repeat the behavior or keep looking for the right answer.

2. Adding in the click makes the trainer more aware of the timing of the release which is important for training using pressure and release.

3. The act of stopping, delivering and allowing the horse to get his reinforcement creates a pause in the training process which gives both horse and trainer time to think.

4. Eating itself (if that is the chosen reinforcement) is reinforcing for most horses and food is an easy reinforcer to use once horses are past the initial stages.

5. Clicker trained horses are looking for the right answer and give excellent feedback to the trainer on how clear they are with their cues and what body language they pick up on. Because they are looking for the answer, they often anticipate and this teaches handlers to have excellent body awareness, body language and teaches the handler a lot about cues.

6. There is an advantage for a trainer in starting with a system they already know as it means they are not learning as many new things at once and they can be more consistent.

7. Because the emphasis is not on "making the horse do it", we give the horse time to respond to our request and we create a willing partner.  

8. The use of positive reinforcement makes the stimulus less aversive, partly through classical conditioning.  This is good in that it makes some horses more tolerant of people when they are learning basic riding skills and mechanics.

    Unfortunately there are some drawbacks too:

1.  If your horse has had bad experiences with a training system, all that emotional baggage is going to still be present and it might make it harder to get the horse interested in clicker training.

2.  Any training system that relies heavily on punishment or escalating negative reinforcement is not going to combine well with clicker training.  In order for clicker training to work well, the horse has to want to participate and a horse that has been punished a lot is not going to want to interact with the trainer or offer behavior.

3.  A trainer who is very familiar with a specific training program might find it is hard to change old habits.

4.  Most training programs do not break behavior down into enough small pieces so you end up relying on capturing behaviors instead of learning how to shape them.

5.  Not only do most training programs have procedures and exercises, but they also have a philosophy too.  If the philosophy is not compatible with clicker training, then it is hard to make the mental shift that needs to happen to fully embrace clicker training.

7.  Using a lot of negative reinforcement does not take advantage of the power of clicker training. It is what Alexandra Kurland calls "sugar coating same old, same old." Yes, it is better than using only negative reinforcement, but it does not necessarily create a thinking partner.   What you get are horses that are good at following directions, but not necessarily horses that are really operant. 

    In the list of drawbacks, I noted that some training programs are not going to combine well with clicker training. These are usually programs that use strong aversives to change behavior, This could be punishment or it could be negative reinforcement that escalates well beyond a horse's tolerance zone.   Adding positive reinforcement to these programs can lead to very mixed results unless the program is modified to be more clicker friendly.   Behavior that is trained by force, scaring the horse, or sending the horse into flight mode is not going to combine well with clicker training because an important component of clicker training is giving the animal freedom to experiment and encouraging the horse to think. 

Poisoned Cues

     The term "poisoned cue" was first used by Jesus Rosales-Ruiz and his graduate students at the University of North Texas when they studied the effects of combining positive and negative reinforcement to train the same behavior.  What they found was that training a new behavior with a combination of negative and positive reinforcement was less effective than using positive reinforcement alone.  While the dog learned the behavior in both cases, the dog's attitude, performance and emotional response to the cue was different when negative reinforcement was used to train it.  This has led to further work on poisoned cues and what happens when you mix positive and negative reinforcement.  I was able to attend sessions on the poisoned cue at Clicker Expo in 2006, 2007 and 2008 and saw the video of the training sessions they presented.  The video clips clearly showed that training a behavior with a combination of positive and negative reinforcement can be problematic.   I do want to note that if they had included training using negative reinforcement alone, it would be easier to see if the cue was poisoned by any use of negative reinforcement or if it was the combination of positive and negative reinforcement that caused the dog's change in behavior and decreased rate of learning.

   In the study they trained a dog to come using only positive reinforcement (a treat) and compared it to training the dog to come using a combination of negative reinforcement (tugging the leash) and positive reinforcement (a treat).   When the cue was trained with only positive reinforcement, the dog would eagerly come up to the trainer when the cue "ven" was given. The dog looked happy and animated with its tail wagging and bouncy steps.   But when the cue "punir" (trained with the combination of positive and negative reinforcement) was given, the dog came with droopy tail and a depressed attitude.   The addition of the treat was not enough to overcome the dog's emotional response to the leash pull.

    In the next part of the experiment, they wanted to see if they could use the cue as a reinforcer. One of the qualities of a cue that has been trained with positive reinforcement is that you should be able to use it to reinforce another behavior.   If the cue is associated with positive reinforcement, then hearing the cue should be reinforcing to the dog because it indicates a chance to earn reinforcement.  So in the next experiment, they did separate trials using either the cue "ven" or the cue "punir."   The cues "ven" and "punir" were used to mark a specific behavior as part of the shaping process or to capture the behavior. The target behaviors were going to a specific location or backstepping.  In this process, instead of clicking, the trainer said either "ven" or "punir" and then clicked and rewarded the dog for coming.  Then they measured to see if the target behavior increased. 

    The results showed that the cue "ven" acted as a reinforcer because the dog started going to the marked square,  but the cue "punir" did not act as a reinforcer because the dog's behavior did not change in any consistent way.  In the trials with "punir," the dog either stayed close to the handler, drifted aimlessly around or tried to offer the behavior that had been reinforced by "ven."  In the trials with "punir," the dog was also very subdued.   This is only the first part of the study, they did some more experiments afterward to further explore using "ven" and "punir" and see if there were other variables that were affecting the dog's response.

     The conclusion presented at the first Clicker Expo where I saw this was that it was the combination of positive and negative reinforcement that poisoned the cue.  Poisoning the cue means that the cue is no longer just a possibility to earn reinforcement. It is also a threat that an aversive is coming.  In a later Clicker Expo, I noticed the wording had changed slightly and it was now a combination of positive reinforcement and an aversive that poisoned a cue.  I noticed this because it meant that it was not any application of negative reinforcement that could poison a cue, it depended upon the aversive. And this makes sense.  Whenever I use negative reinforcement, I run the risk of using positive punishment too.

    This study has sparked a lot of interest among horse clicker trainers because so many of us do use negative reinforcement and it has made us all look more closely at how we use it, which is a good thing.  The first time I heard the presentation, I thought it meant that any combination of positive and negative reinforcement was going to lead to the kind of depressed and ineffective training that the study showed when the cue "punir" was trained. But it turns out that the study was looking at a very specific way of combining positive and negative reinforcement. It was designed to answer the question of what happens if you use positive reinforcement as a consequence for correct answers and negative reinforcement as a consequence for incorrect answers or to "make" the behavior happen, and it was comparing this to what happens if you only reward correct answers.  In simple terms, is it easier to learn something if you are told "yes" and "no" as compared to if you are just told "yes?"  Adding a positive reinforcer for "yes" and a negative reinforcer for "no" were added as consequences to see if they made the difference between "yes" and "no" that much clearer. 

    I do think this is important research and horse people should take note of it because a lot of crossover trainers embrace the idea of adding positive reinforcement, but are reluctant to totally give up the idea of needing to react when the horse is wrong.  They end up falling into a situation where they are rewarding correct responses and reacting with punishment or an aversive to incorrect responses.  But I have to wonder about a few things.  For starters, the dog in the study had no previous experience with leash tugs and its past experience had been with positive reinforcement whereas most horses I meet have already been trained with negative reinforcement.  Most of these horses do very well with the addition of positive reinforcement.  Is adding positive reinforcement to the training of an animal that has been exposed mostly to negative reinforcement different than adding negative reinforcement to the training of an animal that has been exposed mostly to positive reinforcement?   What would happen if you compared positive and negative reinforcement with using negative reinforcement alone? 

    I also want to make a point about the poisoned cue study and how the cues were trained.  In the trials with positive reinforcement only, the trainer said the cue "ven" and if the dog met the criteria, the dog was reinforced with a treat.   In the trials using positive and negative reinforcement, the trainer said "punir" and if the dog met criteria, it was reinforced with a treat.  If the dog did not meet criteria, a leash drag was delivered and this was used to put the dog into position where it was then reinforced with a treat.  The leash drag was used to pull the dog into position where it was then reinforced.  I would say the dog was essentially being "paid" for being pulled into position. I did not see the dog learning that the leash drag meant it could earn reinforcement or learning how to avoid the leash drag. In fact, they changed from a collar attachment to the leash to a harness and then discontinued the leash drags in the "punir" study because they were concerned about harming the dog.  That implies that the leash drag was not working as an effective reinforcer for coming toward the handler. What the leash drag did increase was the behavior of staying by the handler. In later studies, they had trouble getting the dog to go far enough away in order to ask it to come. That makes sense too. The reinforcement was being delivered when the dog was in position near the handler. 

    So putting aside the emotional considerations about training with negative reinforcement, why were the leash drags so ineffective as a training tool?  Is this a problem with negative reinforcement or with this application of negative reinforcement?  From my observations and reading the study, I think the use of  negative reinforcement was unsuccessful for at least two reasons.  One was the timing of the introduction of the verbal cue and the other is the use of the leash.  The leash was used to pull the dog into position.  It was not used to teach the dog to come from gentle pressure on the leash through shaping or successive approximations. The leash was used to pull the dog into the final position. I think this is just a classic case of lumping. If the handler had applied a bit of tension and waited and the released for any movement toward the handler, the results might have been different.

    I also think that using the cue early as part of the shaping process meant the cue was associated with the aversive use of the leash.  If the leash had been used differently to ask the dog for a change instead of to make the dog change, it might not have mattered that the cue preceded the leash drag.  But because the cue was added so early when the behavior was still being learned and the leash was still being used in an aversive way, the cue was permanently associated with the unpleasant part of training the dog to respond to the cue "punir."  The cue became a predictor of punishment not reinforcement.

    I am not writing this to criticize or discount the poisoned cue study which I think is very important.  The reason I am going into this much detail is that I think the poisoned cue study is a great opportunity to study one way to combine positive and negative reinforcement and because it is so well documented, it is easy to look at lots of variables to see what horse trainers want to do differently to avoid getting the same results.  I think it does clearly show how the punishment aspect of negative reinforcement can be so aversive that it overshadows any attempt to soften it by adding positive reinforcement.  I wanted to be clear about the set-up because it shows us ways that we can do things differently and avoid having the same results. 

    I noted earlier that the definition of a poisoned cue had changed slightly from a combination of positive and negative reinforcement to a combination of positive reinforcement and an aversive.  Recently I find that people are referring to poisoned cues as any cue that has been associated with an aversive, punishment or a "correction." I am not sure if this is accurate, but if it is, then the problem with poisoned cues is not the ambiguity of the cue, but just that it has been associated with an aversive. I hope to get more information on this at the next Clicker Expo.   

    I do have some concern about putting the emphasis on poisoned cues on the ambiguity because that would imply that we are better off using positive reinforcement alone or negative reinforcement alone.   Someone asked me once if they were better off just using negative reinforcement so the horse did not feel ambiguous about the cue.  I certainly don't want to start recommending that. I think what we need to do is look more carefully at how we use negative reinforcement in training new cues and in maintaining behavior.  If I want my cues to remain as green lights for reinforcement, then I need to be careful about how I react if the horse does not respond to the cue.  I think the key points to remember about poisoned cues are that they are out there but that we can avoid creating them if we are aware of how we use negative reinforcement in our training.

    I hope there will be more research on poisoned cues, specifically on how to avoid them or how to un-poison them. What I have been told is that the behavior is not poisoned, it is the cue that is poisoned. Jesus Rosales-Ruiz recommends that you just choose a new cue. Do you have to retrain the new behavior to avoid the association with the old cue or can you just add a new cue? He doesn't think so, but I imagine this is one of those areas where it depends. In the initial study, the dog was trained using positive reinforcement and an aversive for about 50 trials. After that, only positive reinforcement was used. Even after 100 or 150 trials, the dog's emotional response to the cue was unchanged. Just removing the negative reinforcement and continuing to use positive reinforcement was not enough.  Would it have worked if they had continued to use positive reinforcement for a longer period of time? I don't know.  But based on that, I think in most cases, it is better to reshape the behavior or choose a new cue rather than try to un-poison the existing one.

    One way to think about this is to think of some things that might be poisoned cues for you.  In his presentations, Dr. Rosales-Ruiz says that for most of us, our name is a poisoned cue.  Thinking about my own name, I can see how the difference between a poisoned cue and a non-poisoned cue can be very subtle.  I respond differently to different people saying my name.    Depending upon who says it, how they say it and my previous experience with what happens after I hear my name, I have different emotional responses.  So it is not just my name itself. It depends upon context.  Using our names as an example also shows why it is so hard to un-poison a cue. 

    Let's say I have a friend, who sometimes phones to share interesting news and sometimes calls to invite me out to do something fun.   Because I sometimes like hearing her voice say my name and I sometimes cringe when I hear her say my name, her voice saying my name is a poisoned cue.  If she changes her ways and now only calls me to invite me out, I might find that my emotional response to her voice changes over time, and if she only calls to invite me out for a LONG time, I might start to feel pretty positive about her calling me. But I am never going to feel the same way as I do about a friend whose phone calls I have always enjoyed.  And if after some period of time, she (the first friend)  calls me up and yells at me,  that ambivalence is going to be back and it might take even longer to get rid of it. Or if this happens enough times, I might find that both emotional responses are muted. I don't have a strong emotional response when she yells at me or when she asks me to do something fun.

    Does it matter when she starts to offer different behaviors (consequences) after I respond to her saying my name? Just thinking about this, I think it might. If she calls me a number of times to invite me out and then calls to yell at me, I am going to feel better about her than if she calls to yell at me the first time and then calls to invite me out.  To be honest, if she calls me back after the first time and I have a choice, I might not even pick up the phone. Even if she calls the next few times to invite me out, my gut reaction when she calls is going to be a negative emotional response. I think of this as being about first impressions and they are very important. I am going to come back to how to create good first impressions later.  In this case, the ambivalence might end up affecting everything about her, so it will go beyond hearing her say my name.

    Back to horses and what you do if you have a poisoned cue. Sometimes I can't change the cue.  If I have a horse has had bad experiences and associates them with a saddle, bridle, halter or some other standard equipment, I might not have the option of choosing a new cue. In that case, I think I would play around with doing things differently.  Horses can be surprisingly particular about the context of things. One solution to a girthy horse, once any real issues have been resolved, is to girth the horse up on the other side.  Tightening the girth on one side triggers anxiety and old responses. Tightening the girth on the other side is fine.  If my horse's halter is a poisoned cue, can I change the halter itself? I don't think we know if that works, but it would be one option to try. Can I change the way I present the halter, the order I do things, or teach my horse another unrelated behavior using the halter so the halter is now associated with a cue to do something fun too.

    Another approach to dealing with a poisoned cue is demonstrated in Alexandra Kurland's DVD "Overcoming Fear and the Power of Cues."  She is working with a horse that has become scared of the saddle because of an accident.   She teaches the horse head lowering and then uses the saddle as a cue for head lowering. The horse learns that by dropping her head, she can make the saddle go away. Once she learns she can control the saddle and that this leads to the saddle moving away and the addition of positive reinforcement, she starts to accept the saddle.  Using the saddle as a cue for a positively reinforced behavior changes the meaning of the saddle from an aversive cue to a more positive cue.  I am not sure if the saddle was a poisoned cue to the horse in the sense that it was ambiguous but it certainly was associated with a negative consequence.

   I think we can all agree that we don't want our horses to find a cue aversive because we want them to associate us and our cues with good things.  Poisoned cues can damage the relationships we have with our horses. One of the reasons many people choose clicker training is because they want the animal to be a willing partner. But beyond this, there are some problems with poisoned cues that are particularly relevant to clicker trainers.

    To understand the significance of poisoned cues for clicker trainers, you have to understand that in clicker training, a cue means that if the animal performs a certain behavior in response to the presented cue, it can earn reinforcement.  A poisoned cue means that when the cue is presented, the animal can earn reinforcement if it does the behavior correctly OR it can expect some kind of aversive if it does not perform the behavior.  Because the cue is no longer just an indicator that something good could happen, the cue itself becomes ambivalent. To the animal, it now predicts either reinforcement or punishment and this means that the animal has a mixed emotional response to the cue. 

    I don't think most horse trainers put as much emphasis into having horses that love to be trained as clicker trainers do.  In traditional training, there does seem to be an attitude that whether or not the horse responds is the first priority and the emotional response is secondary. I am not trying to bad mouth traditional trainers or say they don't want happy animals, but I do see many who have accepted that training horses is about teaching them to do what the trainer wants and that the emotional response to the cue is secondary. Their main concern is if the animal responds to the cue correctly. But for clicker trainers, the emotional response to the cue is very important because we want animals to love cues, we want them to view cues as the doorway to reinforcement and we don't want anything to decrease the enthusiasm our animals have for cues. The more we rely on positive reinforcement, the more important it is to have animals that look for and love cues.  

    One way a poisoned cue can be a problem is in creating a chain or sequence. In a chain, each behavior is being reinforced by the cue for the next behavior.  Being asked to do another behavior means the animal has another opportunity to earn reinforcement.  This only works if the cue itself is reinforcing so that when the animal is cued, it has a positive emotional response to that cue. If the animal does not have a positive emotional response, then the cue is not going to reinforce the previous behavior and the chain or sequence will fall apart.  In addition poisoned cues can lead to problems with reliability, attitude, and the speed of learning.  In the poisoned cue study, not only did the dog show a significant difference in attitude when the poisoned cue was trained, but the dog did not learn the cue as quickly or with as few errors as when it was trained with positive reinforcement only.

     When I first heard about poisoned cues, I was already using Alexandra Kurland's methods which combine negative and positive reinforcement and it made me re-evaluate what I was doing. The good news is that because of how she uses negative reinforcement, Alexandra minimizes the likelihood of creating poisoned cues which means it is possible to train with negative reinforcement without creating them.  This was encouraging to me and by looking at her system and doing some experimenting on my own, I think there are some general guidelines that a trainer can follow to use negative reinforcement in her training without creating poisoned cues or poisoning existing cues.

    If you are interested in learning more about poisoned cues, Alexandra Kurland is working on a DVD that will be on this topic and it should be out in 2009.

How to use negative reinforcement as a clicker trainer

    As a clicker trainer, I use negative reinforcement in three different ways. I use it to prompt or generate behavior, to redirect unwanted behavior, and to maintain behavior.  From a scientific viewpoint, these are all the same thing in that I am increasing a behavior, but I find it useful to recognize that I can be using negative reinforcement to address different aspects of training.  In most teaching situations, I am using it to prompt or generate behavior.  I apply a stimulus, wait for a change and remove the stimulus. I need to be careful about my choice of stimulus to avoid using punishment.  Once a behavior has been learned, or is past the early stages, I am going to be using negative reinforcement to maintain it.  And in some situations I am going to be using negative reinforcement to redirect a horse from doing an unwanted behavior.

     In all these cases, I am choosing to introduce a stimulus so that I can use negative reinforcement to reinforce a desired behavior. It is worth pointing out that there are times when I will use negative reinforcement because there is something about my training situation that allows me to use the removal of an item, or the addition of distance (between the horse and the aversive), to facilitate my training goals.  I can take advantage of negative reinforcement when I am introducing my horse to scary items.  If he stands while I approach with the scary item, he gets a click and treat (positive reinforcement) and I remove the item (negative reinforcement).  If my horse is scared of a fixed object or location, I can reinforce the horse for behaviors I like (approaching it, standing quietly, etc..) by allowing him to put more distance between him and the item as the reinforcement. 

     In this manner negative reinforcement can be used alone or in combination with positive reinforcement to work through fear issues where the emphasis is not on a specific behavior or interaction with the object but more to reinforce acceptance or reduce stress associated with the stimulus. Since this is a different use of negative reinforcement than the main focus of this article, I am not going to go into more details here. I just mention it here because I think it is important to recognize when a powerful negative reinforcer is available, and that taking advantage of it can be part of an effective training strategy.  

    The rest of this article is going to focus on the application of negative reinforcement to generate, redirect, and maintain behavior.  There are similarities in how I use negative reinforcement in all three situations but there are some differences and some choices I need to make to minimize any effects of punishment if that is important to me.   I am going to start by talking about the "aversive" stimulus because that is the common element in all these uses of negative reinforcement.

     Be careful about choosing Aversives

    An aversive is something an animal will work to avoid.  Negative reinforcement often works because the animal is motivated to avoid or remove something that the trainer adds.  There are all levels of aversives. If you go back to the section on negative reinforcement, you can read some of the aversives. Some aversives are just annoying (nagging, loud music, boring class).  Others can be downright painful (headache, loud fire alarm, various kinds of physical contact).  We can all think of some ways in which negative reinforcement is used with horses that makes it similar to punishment. When I tap my horse with a whip to ask it to go forward, am I using negative reinforcement or punishment? Is the horse looking for the right answer or avoiding a consequence?   If I use the whip effectively and with good timing to increase another behavior, then I am using negative reinforcement too, but depending upon how I use the whip, will there be unwanted side effects?

    Luckily for us, most horses will tell us when we are using something that they find aversive and we can use the horse's reactions to help us use negative reinforcement with a minimal use of aversives.  In the poisoned cue study, the dog's whole attitude indicated that there was something unpleasant going on.  They measured which behaviors were increasing and decreasing.  The leash pull that was applied as the negative reinforcer was aversive enough that a lot of other behaviors decreased such as tail wagging and certain types of movement.    They noted that in the trials using "punir," the dog started staying nearer and nearer the handler, trying to avoid or escape the leash pull.  I think that seeing a decrease in a behavior (moving around the room freely) means that in addition to using negative reinforcement to train the dog to come, punishment was going on too.

    This section is titled "be careful about choosing aversives" and I could have said be careful about choosing your stimulus. I prefer to use the word "stimulus" when writing about negative reinforcement because I don't think every application of negative reinforcement uses aversives and aversive is such a loaded word. But I am using the word aversive here because I think that it is important to remember that there are going to be times when the stimulus is an aversive.  It would be nice if we could use negative reinforcement without using aversives at all, but I am not sure that is possible.  There are always going to be some situations in real life where we don't have a lot of options and using an aversive is the best choice for various reasons.  

    I do think that it is possible to keep aversives to a minimum by choosing them carefully, monitoring the horse's response and educating the horse so that it accepts the stimulus as a request for a change in behavior and does not just perceive it as punishment.  If I do use an aversive, I want to be aware of what I am doing and plan it carefully so that I do not end up using a stimulus that is more aversive than necessary.   Any use of stronger aversives is kept to a minimum and if I have to keep using a strong aversive or find myself escalating, then I need to come up with a different training plan. Using one once to get a horse's attention "might" be ok, but I sure don't want to stay there long and I would rather not go there in the first place. 

    How to reduce the aversiveness of the stimulus

    If I want to use negative reinforcement to prompt or initiate behaviors as part of clicker training, then I have to choose my stimulus carefully.  There are lots of ways to approach this problem ,but here are three strategies for choosing a stimulus that is not aversive. One is to choose something that has no meaning and teach the horse to associate it with positive reinforcement. Another is to use the stimulus at a level below that at which the animal finds it aversive, and the other is to desensitize the horse to it through positive reinforcement.  In some ways, these are all the same thing as they are all about taking a stimulus and changing its meaning or association. They are just three different degrees of the same thing. In the first case, the horse has no response to the stimulus so I want to give it meaning. In the second case, the horse is overly sensitive to the stimulus but I can still use it if I am careful. And in the third case, the horse is so sensitive to the stimulus that I can't use it all unless I spend time desensitizing the horse to it.

     In the first case, I can choose something that has no meaning to the horse and use positive reinforcement to give it meaning.  It is hard to find something that has absolutely no meaning, especially once your clicker trained horse catches on to this game. But the idea here is to take something that was neutral and give it a positive association and then use it to train other behaviors.  It becomes a prompt for new behaviors and you can use it in a manner that looks similar to negative reinforcement (where the removal of the stimulus is what the horse is looking for) but in reality you are already using the stimulus as a cue or prompt. In this case, the horse is looking for the removal of the stimulus because that means it is going to get positively reinforced, not because it wants to get rid of the stimulus. 

    I can place my hand gently on the horse and click and remove it a number of times and this shows the horse that the hand has meaning.  Then, with a clickerwise horse, I can start looking for small change. I have done this kind of things for years and just thought of it as a way of letting the horse know the game is on.  But now I realize that is more than that. It is a way to give a stimulus meaning and start out training with the horse in thinking mode from the beginning.

    On the other hand, if I have a stimulus that I really want to use because it seems to be the most useful or appropriate, but my horse finds it aversive, I can often train the horse to respond to it by keeping its use below the level at which the horse finds it aversive.  A horse usually finds something aversive because of the way it is used or applied and by recognizing early signs of discomfort, I can work within the horse's comfort zone.   Using the whip as an example, if I have a horse that is afraid of a whip but I want to use it, I might spend some time exploring my horse's response to the whip.  Is the horse ok if I just hold the whip? Is the horse ok if I move it a little? Can I move it around more or wiggle it?  I can click and reinforce the horse for moving off from a tiny movement of the whip where I have the horse's attention but it is no longer alarmed. The more I reinforce the horse for responding to little changes in the whip's movement, the more the horse will accept it.

    But if I have a horse that is so fearful of the whip, then I have to do more desensitizing.  There are lots of ways to do this and I am not going to go into it here but I could just get the horse used to me carrying the whip and doing nothing. I could teach the horse to associate the whip with positive reinforcement through using it as a target or as an object to interact with in a game such as fetch.  Desensitization can take some time and I find that if a horse is fearful of an object, some of that will go away as the horse learns to trust the handler more and gets more settled because of the positive environment of clicker training. If I have a horse that is really unhappy about me using something, I often just leave it alone for a while and revisit it every now and then to see if anything has changed and the horse is now more ok with it. There are always other ways to train behaviors so if my horse doesn't want me to use something, that's ok.

    How successful this is going to be depends upon several factors and goes back to the idea of poisoned cues. If my horse is a bit unsure about a stimulus but I know it has no previous bad history with it, I will spend time desensitizing the horse or using the stimulus in other situations where it is only associated with positive reinforcement. But if my horse has a lot of anxiety about the stimulus, I am probably going to end up creating poisoned cues because that stimulus is now going to be associated with both positive and negative reinforcement.  It is possible to work through this, but it helps to recognize the problem going in.

    Touch is often used as an example of negative reinforcement and certain kinds of touch can certainly be aversive. But with most animals, there is a level of contact that they will accept. By working within that comfort zone, we can teach the horse that touch is not a bad thing. It is just a cue to do something. If the horse learns that responding to the touch earns reinforcement, the horse's perception of touch will change over time.  The simplest way to think about this is if you want to use a stimulus to train a horse, the horse has to accept it and remain in thinking mode when you use it to prompt new behavior. 

    Alexandra Kurland uses this method in a lot of her exercises. She has the trainer wait the horse out by applying the stimulus at a level below that at which it would normally react and wait.  Even a mild feel on the rope or touch on the side will cause the horse to want to change if you wait long enough. The horse moves or responds and she removes the stimulus. This is still negative reinforcement at work, but the punishment aspect is so diminished that the horses quickly start focusing on what earns the click and the stimulus is not perceived as aversive. By stabilizing and waiting, we give the horse time to respond to our request and we create a thinking horse instead of one who just learns to let us pull her around. 

Learn to recognize signs of tension and anxiety in your horse

    Of course in order to do any of this successfully, you have to be able to tell what your horse finds aversive.  I find that most horses will change their posture, facial expression, or use other horse body language to tell me when they don't like something.  Some common responses are obvious ones such as ear pinning, neck snaking, swishing the tail, kicking out or any kind of threatening or aggressive move. But there are more subtle ones too. Sometimes I just see some tension in the face, wrinkles around the muzzle area or a worried expression. My horse might become slightly high-headed or inverted. The feet might start to move.  If I have a new horse, I can start to pay attention to how the horse is in different situations and I will start to be able to read the stress level and what behaviors are associated with tension and anxiety. 

    Sometimes it is hard to know exactly what is going on and I will just monitor certain things to see if the horse gets better or worse.  For the first few months when I had Stella, she had an odd movement she made with her mouth, a bit like a combination of grinding her teeth and snapping her jaw, but quieter.  I never saw her do it except when we were training and at that point, she would do it when we were free shaping with only positive reinforcement as well as when we were working on other behaviors, so I was pretty sure it was not related to using aversives.   It was a good indicator to me for when she was feeling anxious and as her training progressed, it went away. If it had been associated with any particular cue or behavior, I would have adjusted my training program to see if it went away. 

    There are a lot of things that can cause anxiety or tension in your horse.  It is not always related to your use of negative reinforcement or aversives. You could be triggering an emotional response that is connected to past training.  There could be physical issues. Sometimes it is as simple as asking for too much. If I am training a behavior and the training is not going well, or I am seeing signs of anxiety, I always check to see if I am lumping things too much.  I might ask myself if my rate of reinforcement is high enough. I might check my timing or my presentation of the cue. Am I being clear? I am focusing on the use of the stimulus here because that is often a source of tension or anxiety, but other factors can be important too.

    One way to evaluate how your horse is responding to a combination of positive and negative reinforcement is to mix in some exercises that are based purely on positive reinforcement. I do this on a regular basis and it helps me identify changes in a horse's attitude that might not be obvious if everything I did was a combination of positive and negative reinforcement. I also think it is a nice break and change of pace for the horses.  Mixing in some positive reinforcement only exercises is also good for teaching your horse to think and keeps the horse working toward behavior. If I become overly dependent on using negative reinforcement, I can end up with a horse that is very cooperative and does what I ask, but that is not a full participant in the training process.  In some cases, that is ok.  If you are just trying to improve some aspect of my horse's training and you don't want the horse to be very operant, maybe because it is handled by a lot of different people, or you have not worked a lot on stimulus control, then more emphasis on negative reinforcement at this stage of the horse's training might be more appropriate.

    Doing some exercises with positive reinforcement alone is also a good way to find out if your horse is really ok with something or if he is just  "shut down."  Clicker trainers use the phrase "shut down" to mean an animal that has learned to put up with things it dislikes because it feels it has no choice.  Often I meet horses that are labeled as bomb-proof or good for beginners because they are not bothered by a lot of things that most horses do react to.  In some cases, these horses really are ok, but in other cases, once I show them that they do have choices, I suddenly discover that they do have opinions about things and that they are not ok with everything as I thought. Some people find this disconcerting as their previously well trained horse is now showing some opinions, but in the long run it is a good thing. A lot of these horses are the ones that suddenly 'lose it' for no apparent reason.  Once they start to open up a little bit, you can see early warning signs and that helps to avoid a bigger problem.

Don't insist on using cues the animal finds aversive

    But what if you have a cue that you really want to use, the horse hates it and you can't find a level that you can do that is acceptable to the horse?  There are other options.  When I learned how to ride, I was taught that there were cues for particular behaviors. There are standard cues for riding and I have read books that explain how these cues work by tapping into some physical response on the horse's part.  If you had asked me a few years ago, I would have said it was important to train a horse using standard cues so that other people could ride it and because those cues were chosen for a reason.

    But since then, I have realized that cues are very flexible and that while it is nice if all my riding horses use the same cues, I don't have to get there by training them all the same way.  For example,  when I first started Rosie, she did not like leg cues much and I had a hard time using them at all. She found them annoying and pressure on her side was more likely to make her stop and stomp her foot than move forward. So, rather than use a cue that she found aversive, I used a different cue for forward. Over time I added in my legs and she grew to accept them as part of the cue.   When I wanted to teach her to canter off my outside leg aid, she didn't like that one either. So I taught her to canter off a change in my pelvis and once she knew about cantering and it had a strong reinforcement history, I was able to go back and put it on a standard leg cue.

    This is a very common pattern. I have had a number of cues that the horses disliked at first because they had no meaning for them and then later they accepted the cue readily when it was added to an already trained behavior.  I do have to say that knowing what I know now, I probably could have set up things better so that they accepted the stimulus I wanted to use as a cue, but I still think that in some cases, there are stimuli that are going to trigger unwanted reactions.   The point of this little section is to show that even if your horse finds a stimulus aversive at one time, that doesn't mean it will always be aversive.

    Kathy Sdao talks about how emotions travel backwards from the reinforcement to the marker signal to the cue.   This happens through classical conditioning.  If you are not familiar with classical conditioning, I suggest you do a bit of reading on it.  It is worth learning about as when I am training, I am using both operant and classical conditioning, even if my focus is on operant conditioning. I am not going to explain more about classical conditioning here as it would make this article even longer, and it has been explained well in other places. 

    A reinforcer is something the horse will work to gain.  It could be a reinforcer because it is something the horse wants without any training such as food, release from pressure, or safety and these are usually called primary reinforcers.  Or it could be a reinforcer because I have paired it with something else that is reinforcing to the horse.  When I do this, I am using classical conditioning to transfer the emotions associated with the primary reinforcer to create a secondary reinforcer.  Because of classical conditioning, an event that occurs consistently before a pleasurable event will also become pleasurable. This is how the clicker becomes conditioned. The click becomes associated with the reward because it predicts the reinforcer.  So the click now takes on meaning that it did not have before.  Kathy says that this process can go back another step and if the click is now associated with good things, the cue will become associated with good things and the presentation of the cue will become a positive thing for the animal. 

    I think this means that we have three choices if a horse finds a stimulus aversive but we still want to use it.   The stimulus could be a previously learned cue or a novel stimulus.  As I said in the previous section, depending upon the horses reaction to the stimulus, I can either try to use the stimulus within the horse's comfort zone or I can partly desensitize the horse to the stimulus.  In both cases, I am continuing to use the stimulus at the lowest level possible and reinforcing a lot so that the reinforcement history trumps the horses initial response.  In many cases, this works and the horse will learn to accept the stimulus.  But remember poisoned cues? Using a stimulus that the horse already dislikes means that the early stages of training that behavior are not going to be completely positive for the horse. 

    The third and better option might be to use a different cue and then change cues later.  Then when you add the original cue back in, the animal is so sure what you want and the behavior already has a strong positive association so the transfer of emotion from the reinforcer to the cue occurs much faster. I find that horses will accept previously aversive physical cues when they know now exactly what it means and you can often use a more gentler version than if you were shaping with the cue.

    The next two sections are based on some experimenting I have been doing with my horses.  I was looking for ways to change my application of negative reinforcement to avoid punishment and I found both these strategies useful. The first one is about changing the way you think about training and putting aside some ideas about how we use negative reinforcement.  The second one is about using negative reinforcement in a more pro-active way so that the use of negative reinforcement makes the active use of punishment less likely.

   Thank you for reading. If you have any questions, comments, idea or suggestions, please email me (kabart315@gmail.com.)

Home | Articles | Clicker Basics | Community | FAQ | Getting Started | Horse Stories | Links | Photos | Resources

Equine Clicker Training