Pre-receive hook for rejecting large files

[bryan.gillespie]

Hello,

I would like a simple pre-receive hook that rejects commits containing files larger than a certain size (to force users to use LFS). However, I'm blocked at every turn.

I found this plugin: https://marketplace.atlassian.com/apps/1212934/file-hooks-plugin/version-history

But the "installation instructions" lead to a broken page: https://confluence.atlassian.com/display/UPM/Installing+add-ons?_ga=2.175288618.941315121.1568314065-535602028.1568062230#Installingadd-ons-Installingbyfileupload

I am unable to find that plugin in the "marketplace" like it advises and I can't find anything similar in the marketplace. Note this is for my company's local "BitBucket Server" instance which is really hard to find search results for because all results keep taking me to BitBucket Cloud.

Thanks for any help.

[Mikael Sandberg · Accepted solution]

Hi @bryan.gillespie

The app you found is no longer supported by the developer/vendor so that is why you could not find it in the marketplace. 

There are a couple of apps that you could use, Scriptrunner has a pre hook that allows you to prevent large files being pushed, External Hooks and Javascript Hooks allows you to write your own hooks.

[Mikael Sandberg]

Another option that would not require an app would be to create your own client side hook and store it in a repository, see Including Hooks in a Git Repository. The drawback is that you have to make sure each client is configured to use the hooks.

[Julius Davies [bit-booster.com]]

The reason the "installation instructions" lead to a broken page is probably because that particular app is not compatible with Bitbucket 6.x.  According to the marketplace page you linked to, its most recent compatibility range was:

Bitbucket Server 5.2.0 - 5.16.10

p.s. On October 8th, 2019, we (bit-booster) released a new version of our free Control Freak plugin that now includes the ability to block large files from being pushed into Bitbucket Server or Data Center.  See our separate answer on this thread below for more information.

[bryan.gillespie]

Ah, I see. Thanks for the advice. I will look into those options. Maybe I'll create a new app for the marketplace that does this.

[bryan.gillespie]

Thanks @Julius Davies [bit-booster.com] ! Control Freak is just what I was looking for.

[nuetzig]

For client-side hooks use https://github.com/gabyx/githooks
For server-side this can be used as well.

[Julius Davies [bit-booster.com]]

Our free Control-Freak plugin now offers this functionality (compatible with Bitbucket Server and Data Center 5.8.0 - 6.x).

Here's a screenshot:

Here's what the rejection message looks like:

$ git push
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (5/5), 7.46 MiB | 4.47 MiB/s, done.
Total 5 (delta 1), reused 0 (delta 0)
remote: Control Freak: Push rejected. Files larger than 8000000 bytes are not permitted
remote: 
remote: Control Freak
remote: -----
remote: Push rejected. Files larger than 8000000 bytes are not permitted
remote: in this repository. The following file is too large:
remote: 
remote: path/to/bigFile.jar (8722752 bytes)
remote: 
remote: You can push large files to this repository using 'git lfs':
remote: git lfs track <path>
remote: 
To ssh://vm.bit-booster.com:7999/bb/rep_1.git
! [remote rejected] master -> master (pre-receive hook declined)

[Daniel Trezub]

I have a question about this: If I'm trying to push a large file, will the file be uploaded and THEN processed, or is the git client telling the server "hey, I have a large file to upload, may I?" and the server will stop the file from going through the network if the check fails?

Thank you!

[PeteToscano]

Bryan:

I think we were in a similar situation. I believe we were using the same plugin, too. We had to find alternatives in order to stay current with Bitbucket Server. 

I installed Adaptavist Script Runner for Bitbucket Server. The initial install was for a different function, which it does very well. I thought I lucked out with the "restrict file size" pre-hook that it has, but it turns out that, if the commit contains a file of the type we want to limit (PDF), the restriction will apply to _any_ file in that commit, not just the PDFs. This can be worked around, but I'd rather my users not have to think about this.

I tried to play around with Script Runner more, but I have no Groovy experience and got lost quickly, so now I'm back to considering writing my own script and putting it on the server. This isn't too terrible, but I can't help thinking that I'm reinventing the wheel.

If you have any suggestions or even dead ends, I'd love to hear them.

Thanks,
Pete

[Julius Davies [bit-booster.com]]

Pete,

I could add an additional filter to my free Control-Freak plugin.  Right now it does this:

[x] - Block large files. Maximum allowed file size (bytes): [     ]

[x] - Encourage Git LFS in push rejection message.

I *could* add the following control:

[x] - Only block large files that end in the following extensions: [ <comma-separated> ] (e.g., *.zip, *.exe, *.pdf, *.jar)

Is that what you need?  Note: files that fall below (or equal) to the threshold would never be blocked, even if they had the bad file extension. Set the threshold to zero to completely ban that file type.

[PeteToscano]

Yes, I believe that it would. 

[Julius Davies [bit-booster.com]]

Okay!  Done!  Look for v2019.10.20 of Control Freak in marketplace.atlassian.com.

Screenshot:

Consequence for a push with large files:

remote: Control Freak
remote: -----
remote: Push rejected.
remote: All files larger than 10 bytes are not permitted.
remote: Specific files [*.exe, *.pdf, *.zip] larger than 5 bytes are also not permitted.
remote: The following files are too large:
remote: 
remote: binary/f.exe (7 bytes)
remote: binary/g.zip (7 bytes)
remote: 
remote: You can push large files to this repository using 'git lfs':
remote: git lfs track <path>
remote: 

[PeteToscano]

Very nice. Should issues with my hand-written pre-receive hook pop up, I'll be sure to give Control Freak a look. 

[Jean-Pol Landrain]

It is very nice, but if I can suggest an improvement: it is possible to disable Git LFS in the Bitbucket server settings. Can you also take the value of this option into account in order to disable the part of the message that suggests to use the "git lfs" command ? The option should be kept in Control Freak, but it shouldn't be taken into account if Git LFS is disabled in the server settings.

[Anoop Wilson]

I am unable to block specific file types from pushing to the central repository.

Though there's an option to block certain file types as below. But it's not working at all.

I tried with *.pdf and pdf and .pdf conventions. But nothing is working. Still, I am able to commit pdf files to the repository without any trouble.

But this plugin is able to block larger files but not specific file types.

Any alternate plugin, incase if this plugin can't do this job ?

Thanks in advance.

[Jean-Pol Landrain]

I have done a quick test and unfortunately I confirm it's not working for me either (version 2020.02.07)

[Julius Davies [bit-booster.com]]

Are you sure?  This is working for me (I'm the primary maintainer of this plugin):

remote: Control Freak: Push rejected.
remote: 
remote: Control Freak
remote: -----
remote: Push rejected.
remote: All files larger than 500 bytes are not permitted.
remote: Specific files [*.exe, *.pdf, *.zip] larger than 200 bytes are also not permitted.
remote: The following file is too large:
remote: 
remote: test.pdf (1723760 bytes)
remote: 
remote: You can push large files to this repository using 'git lfs':
remote: git lfs track <path>
remote: 

There is one important loophole to know about with this control:  any file that already exists in the repo (on any branch or tag, including unreachable tags or branches, and including files deleted many commits ago - they are still present in the repo) cannot be blocked (based on contents, not based on filename).  The control only gets triggered during the phase where git-receive is calculating missing files to accept during the push, and since git-receive skips files that already exist on the server, regardless of their name, there is no way to block those.

This means developers can also easily get around the control:  "git push random-file.allowed" and then rename to "random-file.pdf" in a later push.

Is there any chance you guys are hitting against this loophole, where byte-for-byte identical files (regardless of name) cannot be blocked if they are already present in bitbucket?

[Julius Davies [bit-booster.com]]

p.s.  the control happily accepts all of these:  pdf, .pdf, *.pdf

(they all evaluate to the same file type)

[Jean-Pol Landrain]

I have done a few more tests this morning and in my case it's possible I was into this loophole in my previous tests.

Please, note however that if you don't specify a maximum size (it is not very clear that it is a mandatory field), I believe it should take a size of '0' bytes as default. I don't know if there is a default size at the moment, but it doesn't seem to block anything unless you specify the size (also there is just one pattern both for the filename and for the size; this limits the possibilities).

You say above that all the pdf, .pdf, and *.pdf evaluate to the same file type. Could you also indicate if it is possible to use more complex patterns or even regular expressions ? Thank you.

I know this addition is not very old in the Control Freak plugin, and I just try to suggest improvements. This is not criticism, only suggestions.

Thank you very much for your work on this feature, and on this plugin in general.

[Anoop Wilson]

Any plugin that can substitute File hooks plugin for bitbucket 6.x Datacenter?

A control freak plugin can block for the entire server. But we are interested in project-specific blocking. Is that possible with a control freak plugin. Because every project has it's own limitations and policies.

Anyone please suggest an alternative plugin to File hooks plugin which can reject the commits of files of a certain size and file name of certain pattern.?

Thanks in advance.

Anoop

[Jean-Pol Landrain]

Yes, you can use Control Freak. It is very well done: you have configuration options for the whole Bitbucket instance, but all the settings can be overriden both at the Git project level and at the repository level. So, it allows for very fine tuned settings.

[Levente Szabo _Midori_]

Another way you can go about solving this is with Better Commit Policy for Bitbucket.  (Please note that relies on Better Commit Policy for Jira!) You can also be more effective with the local hooks, provided by the app. By using local hooks, you can catch non-compliant commits right on the developer's machine at commit time, so it's guaranteed to pass when pushed to the server.